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v>  Abstract 

N.. 

Currently,  there  is  no  in-residence  technical  training 
course  for  Supply  warehousemen,  AFSC  645X1.  Therefore,  base 
level  supervisors  and  trainers  are  responsible  to  provide 
the  initial  technical  training  for  direct  duty  assigned 
airmen  in  the  645X1  AFSC.  Given  the  absence  of  a  formal 
technical  training  course,  most  base  level  supply  squadrons 
use  the  64531  Career  Development  Course  (text)  to  train 
their  direct  duty  assigned  warehousemen.  The  purpose  of 
this  study  was  to  conduct  an  experiment  which  compares  the 
effectiveness  of^Computer-Based  Training  ( CBT rJwiT’w 
commonly  used  text  training,  using  a  CBT  module  which  waB 
developed  by  the  new  supply  CBT  development  team,  located  at 
Lowry. AFB,  CO.  The  results  of  this  B^SSSp'-nave  shown  that 
when  used  properly,  CBT  can  increase  the  amount  of  learning 
which  takes  place,  increase  the  ability  of  trainees  to 
perform  the  tasks  for  which  they  are  trained,  and  reduce 
total  training  time.  This  research  directly  supports  the 
Air  Forces'  continued  use  of  CBT  for  the  initial  training  of 
supply  warehousemen,  and  further  suggests  that  CBT  may  be  a 
suitable  technique  for  other  Air  Force  training  needs. 
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EVALUATION  OF  THE  EFFECTIVENESS  OF  THE 
STORAGE  AND  DISTRIBUTION  ENTRY-LEVEL 
COMPUTER-BASED  TRAINING  ( CBT)  PROGRAM 

!_*.  Introduction 

General  Issue 

In  November,  1985  HQ  USAF/DPPE  and  the  Air  Training 
Command  (ATC)  made  a  joint  decision  to  eliminate  the  initial 
Supply  Materiel,  Storage  and  Distribution  Specialist 
Training  Course,  Air  Force  Specialty  Code  (AFSC)  645X1,  at 
the  3440th  Technical  Training  Center  (TTC),  Lowry  Air  Force 
Base,  Colorado.  The  last  students  graduated  from  this 
course  in  September  1986  (46:  Attachment  6-4).  New  Airmen 
with  the  645X1  AFSC  are  now  sent  Direct  Duty  Assignment 
(DDA)  from  basic  training  to  base  level  supply  squadrons, 
and  the  initial  training  burden  is  now  being  placed  on  the 
base  supply  training  section  and  the  trainees'  supervisors 
(46:  Attachment  6-5). 

In  November  1988,  ATC  and  the  Air  Force  Logistics 
Management  Center  (AFLMC)  created  a  CBT  development  office 
at  the  3440th  TTC.  The  goal  of  this  office  is  to  develop 
CBT  programs  which  will  provide  the  initial  training  for 
supply  warehousemen,  AFSC  645X1.  The  purpose  of  the  CBT 
programs  is  to  teach  DDA  Air  Force  supply  personnel  (645X1) 
basic  supply-related  procedures  at  the  base  level.  The 
Storage  and  Distribution  Entry-Level  CBT  Program,  which  is 
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used  in  this  study,  was  scheduled  for  completion  in  December 
1989  (46:  Attachment  6-6).  Approximately  20  other  supply 
related  programs  (lessons)  are  presently  undergoing  final 
revisions,  and  should  be  released  to  the  field  by  October 
1990. 

Specific  Problem 

Research  should  be  conducted  to  determine  the 
effectiveness  of  the  Materiel,  Storage  and  Distribution 
Entry-Level  CBT  programs  before  they  are  released  for  use 
throughout  the  Air  Force.  The  term  "effectiveness”  refers 
to  the  degree  to  which  learning  takes  place,  the  time 
required  to  complete  the  training,  and  the  ability  to 
perform  an  actual  task. 

Hypothesis 

The  Storage  and  Distribution  Entry-Level  CBT  Program  is 
a  more  effective  training  tool  for  training  DDA  645X1 
personnel  than  is  the  64531  Career  Development  Course  (CDC). 

Investigative  Questions 

In  order  to  better  determine  whether  to  reject  or 
accept  the  hypothesis,  research  must  be  conducted  to  answer 
the  following  investigative  questions: 

1.  What  are  the  present  base  level  training  methods 
being  used  to  train  DDA  supply  warehousemen,  AFSC  645X1? 
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2.  Do  the  AFSC  645X0  experimental  teat  subjects 
represent  a  statistically  similar  population  when  compared 
to  newly  assigned  DDA  645Xls? 

3.  Are  there  any  significant  differences  between  the 
test  subjects'  AQE  scores  from  the  three  experimental 
treatments? 

4.  Do  the  test  subjects  learn  more,  as  measured  by  a 
post-test,  when  trained  using  CBT  versus  CDCs? 

5.  Do  the  subjects  perform  the  actual  task  better  when 
trained  using  CBT? 

6.  Is  training  time  reduced  when  the  test  subjects  are 
trained  using  CBT? 

7.  Does  computer  experience  and  education  affect  the 
student's  comprehension  of  the  material  presented  via  CBT? 

Terms .Defined 

The  following  terms  are  defined  for  reader  clarity. 

1.  AFSC  645X1  -  Materiel,  Storage  and  Distribution 
Specialist.  Individual  is  responsible  for  the  accurate  and 
timely  receipt,  storage,  inventory,  issue,  delivery,  and 
shipment  of  property  (21:  Attachment  37). 

2.  AFSC  645X0  -  Inventory  Management  Specialist. 
IndivJdual  computes  requirements,  allowances,  and  performs 
research  on  supplies  and  equipment.  Individual  is  also 
responsible  for  inventory  stock  control  (21:  Attachment  37). 

3.  Computer  Assisted  Instruction  (CAI)  -  This  term  is 
used  when  a  computer  is  utilized  to  administer  training  to 
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users.  This  training  should  provide  interaction  between  the 
user  and  the  computer  (37:14). 

4.  Computer  Managed  Instruction  (CMI)  -  This  term  is 
used  when  a  trainer  uses  the  computer  to  plan  training 
programs,  record  student  performance,  or  assign  lessons 
(56). 

5.  Computer  Based  Training  (CBT)  -  Using  a  computer  to 
implement  and  manage  training  (37:17).  The  term  CBT  will  be 
used  throughout  this  thesis  to  refer  to  any  training 
programs  utilizing  a  computer. 

Background 

The  use  of  CBT  has  become  commonplace  in  the  private 
sector.  A  survey  of  200  randomly  selected  industrial 
training  managers  conducted  by  Kearsley  and  Hillelsohn 
showed  that  forty- two  percent  of  managers  were  using 
computers  in  training  activities  and  another  forty-one 
percent  were  planning  on  implementing  computers  in  training 
(42:22).  CBT  is  also  UBed  extensively  in  the  public  sector. 
Dr.  Alfred  Bork,  Director  of  the  Educational  Technology 
Center  at  the  University  of  California,  predicts  that 
"computers  will  comprise  the  dominate  delivery  system  in 
education  for  almost  all  age  levels  in  most  subject  areas" 
over  the  next  25  years  (11:4).  Limited  use  of  CBT  has  taken 
place  in  the  Armed  Forces.  The  Navy  has  taken  the  lead  in 
the  use  of  CBT.  At  the  present  time,  approximately  12.6 
percent  of  the  Navy's  courses  with  large  student  enrollment 
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use  some  type  of  CBT,  and  entry  level  courses  use  CBT 
programs  much  more  often  than  advanced  courses  (70:16). 

Many  studies  have  been  conducted  to  investigate  the 
learning  and  cost  effectiveness  of  CBT.  In  the  majority  of 
these  experiments,  CBT  provided  positive  results  in  terms  of 
learning,  and  in  cost  savings  associated  with  the  reduction 
in  training  time  and  better  utilization  of  training 
personnel  (20:42;  28:64;  29:14;  70:40).  Even  though  CBT 
appears  to  be  very  effective  in  many  training  situations, 

CBT  should  not  be  forced  into  an  area  of  training  where  it 
may  not  be  applicable,  or  when  other  techniques  may  be  more 
suitable.  A  need  should  be  identJ 'led  before  CBT  is 
introduced.  CBT  should  only  be  implemented  when  the  reasons 
why  the  computer  will  help  training  can  be  identified 
(70:20).  In  most  instances,  CBT  Bhould  not  be  used  as  a 
stand  alone  training  program.  "CBT  should  be  used  as  an 
ancillary  tool  or  aid  for  the  instructor  where  it  can  make  a 
significant  contribution  rather  than  replacing  the 
instructor"  (70:20). 

In  September  1986,  the  last  students  graduated  from  the 
645X1  Initial  Materiel,  Storage  and  Distribution  Apprentice 
Training  Course  at  Lowry  AFB.  Since  its  elimination,  no 
course  has  been  developed  to  fill  this  training  void 
(46 attachment  6-4).  The  results  of  eliminating  this 
training  course  are: 
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1.  Large  numbers  of  untrained  personnel  have  been 


assigned  directly  to  the  field. 

2.  A  heavy  training  requirement  has  been  inherited  by 
base  supply  training  sections  and  the  trainees'  supervisors. 

3.  The  large  influx  of  untrained  personnel  has  reduced 
trainee  competency  levels,  negatively  effecting  mission 
accomplishment  (46:  Attachment  6-5). 

In  October  1988,  approval  was  granted  for  ATC  and  the 
AFLMC  to  develop  and  produce  CBT  lessons  for  initial  base 
level  training  for  DDA  645X1  personnel.  Development  of 
these  lessons  began  in  November  1988.  The  cost  of  producing 
the  lessons  was  estimated  at  $300,000,  which  was  to  be 
spread  over  the  18  month  development  period  (46:  Attachment 
6-6).  The  lessons  were  developed  as  an  on-the-job  training 
(OJT)  enhancement  program  (46:1). 

At  the  22-23  February  1989  Supply,  Computer-Based 
Training  Tiger  Team  Meeting,  Lt  Col  Tim  0.  Peterson,  then 
Director  of  Supply,  AFLMC,  referenced  an  October  1988  study 
of  the  benefits  Fortune  1000  Companies  realized  using  CBT. 
These  benefits  are  (46:  Attachment  6-19): 

1.  Improved  training  availability 

2.  Individualized  training 

3.  More  cost-effective 

4.  Frees  supervisors  from  routine  training 

5.  Improves  transfer  of  knowledge  to  the  task 
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The  widespread  use  of  CBT  in  the  private  sector, 
education,  and  in  a  limited  scope  in  the  military,  indicates 
CBT  is  a  valuable  medium  in  some  training  environments. 
Therefore,  the  potential  use  of  CBT  should  be  explored 
carefully  in  order  to  bolster  training  effectiveness.  Since 
the  elimination  of  the  645X1  Initial  Materiel,  Storage  and 
Distribution  training  course  at  Lowry  AFB  has  created  a 
training  void,  the  use  of  CBT  to  fill  this  void  should  be 
thoroughly  evaluated. 

Before  implementing  a  CBT  training  module  to  fill  the 
645X1  training  void,  the  CBT  should  be  tested  to  insure  that 
it  will  supply  effective  and  efficient  training.  The 
experimental  design  used  in  this  thesis  objectively 
evaluates  the  effectiveness  of  the  CBT  program  developed  and 
produced  by  ATC  and  AFLMC. 

SCQ.Bg 

Research  was  performed  to  satisfy  two  objectives.  The 
first  objective  was  to  provide  the  reader  with  an 
understanding  of  how  CBT  evolved,  the  intricacies  involved 
when  applying  CBT  to  a  training  situation,  the  current  and 
future  status  of  CBT  development  in  the  Air  Force,  and 
current  learning  theory.  The  second  objective  was  to  design 
an  experiment  which  will  determine  how  effective  CBT  is 
compared  to  another  currently  used  training  method. 

The  first  objective  will  be  satisfied  through  the 
literature  review.  Sources  consist  of  periodicals. 
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technical  journals,  telephone  interviews,  personal 
interviews,  government  reports,  theses,  dissertations, 
professional  conference  seminars,  and  Air  Force  Regulations. 
A  brief  history  of  CBT  is  presented  in  order  to  provide  an 
understanding  of  how  CBT  evolved,  and  the  advantages  and 
disadvantages  of  implementing  CBT  for  instruction  of 
personnel  will  be  investigated.  The  results  of  previous 
experiments  will  be  studied  to  determine  if  CBT  is  useful  in 
a  large  variety  of  training  situations  or  applicable  only  to 
narrowly  defined  training  situations.  Research  also 
includes  the  exploration  of  the  traditional  training  methods 
used  in  the  training  environment.  The  authoring  system  used 
to  develop  the  Storage  and  Distribution  Entry-Level  Training 
Program  (QUEST,  version  3.0)  will  be  described  in  order  to 
determine  the  reasons  why  ATC  and  AFLMC  chose  to  use  QUEST 
versus  other  available  systems.  Current  theories  and 
principles  of  learning  are  addressed  to  give  the  reader  an 
understanding  of  the  logic  behind  CBT  development.  Also, 
the  concept  of  training  is  discussed  to  provide  the  reader 
with  an  appreciation  of  the  importance  of  training  and  the 
need  to  develop  training  which  '‘transfers"  easily  to  the 
workplace . 

In  order  to  satisfy  the  second  objective,  research  was 
conducted  to  determine  the  best  experimental  design  in  order 
to  ascertain  if  the  supply  CBT  program  is  more  effective 
than  current  training  methods.  The  experimental  design 
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compares  the  effectiveness  of  the  645X1  CDC,  which  is 
currently  being  used  by  most  base  level  supply  squadrons, 
and  the  Storage  and  Distribution  Entry-Level  CBT  Program  as 
administered  to  645X0  personnel  in  Entry-Level  Training  at 
Lowry  AFB. 

Specific  conclusions  about  this  research  project  will 
be  made  in  reference  to  the  effectiveness  of  the  Entry-Level 
Storage  and  Issue  CBT  lesson,  used  for  the  initial  training 
of  DDA  645X1  personnel.  The  results  should  be  generalizable 
to  all  CBT  lessons  created  at  Lowry  AFB,  since  the  other 
supply-related  lessons  were  created  and  developed  using  the 
same  learning  concepts  and  instructional  techniques. 

Limitations  of  the  Study 

All  research  projects  possess  limitations  in  one  form 
or  another.  Since  researchers  cannot  completely  control  all 
of  the  variables  affecting  their  research,  it  is  important 
that  limiting  factors  which  affect  the  research  be 
identified  and  understood.  The  following  limitations  are 
identified  as  having  a  possible  effect  on  this  research 
project: 

1.  Since  supply  warehousemen,  AFSC  645X1,  are  sent 
directly  from  basic  training  to  their  duty  assignments,  it 
is  very  difficult  to  get  a  large  number  of  newly  assigned 
645Xls  together  for  experimental  purposes.  Therefore, 
entry-level  supply  operations  personnel,  AFSC  645X0,  were 
used  as  test  subjects  for  this  research. 


9 


The  extent  of  this  limitation  will  be  determined  by 
comparing  the  Airman  Qualification  Examination  (AQE)  test 
scores  in  the  general  category  of  the  645X0  experimental 
subjects  to  those  of  645X1  personnel.  These  AQE  scores  will 
be  obtained  from  the  Human  Resources  Laboratory  Data  Bank 
(HRLDB),  located  at  Brooks  Air  Force  Base,  Texas. 

2.  The  relatively  small  total  sample  size  may  not 
provide  data  which  is  truly  indicative  of  the  645X1 
population.  During  the  early  phases  of  this  research,  the 
3440th  TTC  had  1-3  Basic  Supply,  AFSC  645X0,  classes  begin 
each  week  with  class  sizes  ranging  from  12-25  students. 

Since  data-gathering  trips  were  scheduled  during  weeks  when 
3  classes  were  scheduled  to  begin,  an  estimated  40-75 
experimental  subjects  could  be  tested  during  one  week. 

Since  the  authors  planned  to  make  4  experimental  trips  to 
Lowry  AFB,  a  total  sample  size  of  160-300  subjects  was 
originally  projected. 

However,  recent  cutbacks  in  the  number  and  size  of 
classes  have  resulted  in  a  much  smaller  total  sample  size. 
Currently,  an  average  of  1  new  class  begins  each  week,  with 
7-12  students  in  each  class.  Given  that  the  authors  made  4 
experimental  trips  to  Lowry  AFB,  a  total  sample  size  of 
28-48  subjects  was  expected  (a  total  of  49  subjects  was 
actually  obtained).  Therefore,  the  authors  were  forced  to 
accept  a  total  sample  size  which  was  much  smaller  than 
originally  planned. 
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3.  Coordination  and  planning  at  Lowry  AFB  for  the  data 
gathering  trips  was  performed  for  the  authors  by  a  point  of 
contact  (POC)  at  Lowry  AFB.  The  POC  was  responsible  for 
contacting  and  coordinating  with  several  organizations  at 
Lowry  AFB,  to  prepare  for  the  experimentation.  Coordinating 
through  the  POC  resulted  in  some  problems  which  the  authors 
could  have  handled  had  they  been  at  Lowry  AFB  prior  to  the 
initial  experimentation.  It  is  important  to  note  that  these 
problems  did  not  arise  due  to  the  lack  of  effort  on  the  part 
of  the  POC,  but  resulted  from  a  lack  of  understanding  of  the 
intricacies  and  complexities  of  this  experiment. 

4.  The  post-test  is  another  possible  limiting  factor. 
It  was  designed  to  test  the  specific  objectives  used  by  the 
CBT  module,  which  was  used  in  this  experiment.  The  text 
module  includes  excerpts  from  the  64531  CDC,  which 
correspond  with  these  same  objectives.  Therefore,  it  is 
possible  that  the  post-test  may  favor  the  test  subjects  in 
the  CBT  treatment.  To  better  analyze  this  possible  limiting 
factor,  a  task  analysis  is  incorporated  into  the 
experimental  design.  The  task  analysis  is  a  neutral 
measurement  tool  which  does  not  favor  either  treatment.  It 
simply  measures  the  ability  of  the  test  subjects  to  perform 
the  task  of  properly  warehousing  a  piece  of  property. 

5.  Another  possible  limitation,  which  may  effect  the 
results  of  this  study,  is  that  each  of  the  authors  traveled 
to  Lowry  AFB,  and  performed  this  experiment  on  different 
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occasions.  Therefore,  there  is  a  possibility  that  the 
authors  may  have  unintentionally  given  their  respective  test 
subjects  a  different  set  of  expectations  based  on 
differences  in  their  personal  interactions.  Obviously,  it 
would  be  ideal  for  the  same  person  to  present  the  experiment 
to  the  test  subjects  each  time,  ensuring  that  interactions 
with  each  treatment  were  always  identical. 

Since  it  was  impossible  for  only  one  person  to  travel 
to  Lowry  AFB  on  four  separate  occasions,  the  authors  went  to 
great  lengths  to  insure  that  their  interactions  with  the 
test  subjects  were  identical.  As  you  will  see  in  Chapter 
IV,  the  interactions  with  the  test  subjects  were  carefully 
planned,  to  the  point  of  using  written  introductions. 
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II.  Literature  Review 


Overview 

This  literature  review  will  present  a  brief  history  of 
CBT  along  with  the  advantages  and  disadvantages  of  using  CBT 
in  training  situations.  The  results  of  experiments  on  CBT 
programs  in  various  training  environments  will  be  presented 
to  determine  if  CBT  is  effective  in  different  training 
environments.  Also,  the  current  and  future  status  of  CBT 
development  in  the  Air  Force  will  be  described,  as  will  the 
traditional  training  methods  used  in  various  training 
environments,  and  the  current  theories  and  concepts  of 
learning.  Finally,  the  features  and  capabilities  of  the 
QUEST  authoring  system  used  to  produce  the  Entry-Level 
Storage  and  Issue  CBT  program  will  be  highlighted. 

History  ol  CEE 

This  section  of  the  literature  review  provides  a  brief 
description  of  the  historical  development  of  CBT.  This 
background  will  provide  the  reader  with  valuable  background 
information,  and  an  understanding  of  how  and  why  CBT  was 
developed. 

In  1926,  Sidney  Pressey,  an  Ohio  State  University 

professor,  developed  a  machine  named  the  "Pressey  Testing 

Machine".  This  machine  taught  and  tested  students  (61:103). 

A  student  studied  the  subject  in  the  usual  way  and  then 
turned  to  the  machine.  The  machine  directed  the 
student  to  a  first  item  on  a  multiple-choice  test,  and 
the  student  made  a  choice  by  pressing  a  numbered  key. 

If  the  choice  was  right,  the  machine  moved  on  to  the 
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next  item;  if  the  choice  was  wrong,  the  student 
pressed  another  key.  When  the  student  went  through  the 
test  a  second  time,  the  machine  stopped  only  on  those 
items  on  which  the  student's  first  choice  had  been 
wrong.  (61: 103; 104) 

The  next  major  innovation  in  the  use  of  computers  in 

education  and  training  occurred  in  the  1950s.  B.  F. 

Skinner,  professor  emeritus  from  Harvard  University, 

developed  a  machine  that  was  an  improvement  over  the  earlier 

machines;  Skinners'  machine  taught  the  students  the 

material.  The  items  on  the  testing  portion  were 

sequentially  arranged,  so  that  if  the  student  answered  the 

question  correctly  in  the  first  frame  he  was  better  prepared 

to  move  to  the  next  question  (37:9).  SkinnerB'  machine  was 

fashioned  directly  after  his  theory  of  learning. 

Skinner's  analysis  of  instruction  assumes  that 
motivation  must  be  present,  that  the  students  muBt 
make  a  response,  and  that  this  response  needs  to  have 
consequences  which  are  reinforcing.  The  increased 
specificity  of  Skinner's  suggestions  center  around  the 
principle  of  stimulus  control.  (3:18) 

The  interaction  between  the  student  and  the  computer 

was  termed  "computer-based  interaction".  Until  this  point 

in  time,  CAI  was  seen  as  possibly  replacing  teachers.  But 

in  the  1960s,  International  Business  Machines  (IBM)  declared 

that  these  systems  were  a  support  tool  for  the  teacher  to 

use,  not  a  replacement  for  teachers  (37:11).  In  the  early 

1960s,  IBM  developed  Coursewriter  I,  the  CAI  authoring 

language.  Soon  after  the  introduction  of  Coursewriter  I, 

many  other  authoring  systems  were  introduced  into  the 

market.  The  availability  of  authoring  systems  and  the 
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increased  capabilities  of  computers  in  the  mid-1970s  led 
many  educators  to  believe  CAI  would  play  a  large  role  in  the 
future  of  education  (6:1).  However,  others  did  not  agree 
and  were  very  skeptical;  they  believed  that  CAI  could  not 
motivate  and  provide  necessary  interaction  between  students 
and  teachers.  Critics  said  computers  could  never  come  close 
to  replacing  teachers.  Now,  fifteen  years  later,  the  same 
people  are  going  into  classrooms  and  observing  how  computers 
have  revolutionized  learning  in  the  classroom  (32:167). 

In  1980,  approximately  22,000  public  schools  provided 
CAI ,  which  represents  nearly  50  percent  of  all  secondary 
schools  and  14  percent  of  all  elementary  schools  (6:1).  In 
1983,  there  were  291,000  computers  available  for 
instructional  use  in  the  public  school  system.  There  is  no 
indication  that  the  upward  trend  in  the  use  of  computer 
instruction  will  decline  in  the  future  (67:62). 

Methodology /Design  ol  CBT 

Perhaps  the  greatest  strength  of  CBT  is  the  flexibility 
it  can  provide  trainers.  CBT  can  be  used  to  meet  many 
training  objectives.  In  order  to  meet  varying  objectives 
different  CBT  methodologies  need  to  be  utilized.  The 
different  methodologies  consist  of  drill  and  practice, 
tutorials,  simulations,  and  tests  (37:17).  This  section  of 
the  literature  review  will  investigate  the  structure  and 
logic  behind  each  of  the  aforementioned  CBT  methodologies. 
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Drill  and  Practice .  The  CBT  drill  and  practice  method 
is  most  effective  in  the  training  of  routine  skills.  Drill 
and  practice  presents  the  item  more  than  once  in  order  to 
increase  fluency,  speed,  and  retention  of  the  subject 
matter.  If  the  student  answers  incorrectly,  the  item  will 
be  presented  more  often.  This  method  should  be  used  only  to 
reinforce  learning  of  material  already  presented  to  the 
student  (37:18;  68:43-44). 

There  are  five  steps  in  the  drill  and  practice 
procedure  (37:18;  68:46): 

1.  An  introductory  page  is  presented  to  the  student. 
The  page  should  consist  of  a  statement  of  learning 
objectives,  and  directions  on  how  to  do  the  lesson. 

2.  The  student  selects  an  item.  The  selection  can  be 
a  question,  problem,  or  some  other  form  of  exercise. 

3.  The  student  formulates  a  response  and  inputs  the 
response  using  the  computer  keyboard  or  other  input  device. 

4.  The  program  judges  the  response. 

5.  The  program  provides  feedback  to  the  student. 

The  entire  5  step  procedure  is  repeated  until  the 

student  decides  to  quit,  or  the  student  has  satisfied  the 
requirements  of  the  Drill  and  Practice  program  (37:18). 

Tutorials.  A  tutorial  is  very  similar  to  drill  and 
practice,  but  unlike  drill  and  practice  the  tutorial  can 
stand  alone  as  a  CBT  program.  The  tutorial  presents  new 
information  to  the  student  and  then  evaluates  the  student's 
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comprehension  of  the  material  (37:19).  The  tutorial  also 
provides  the  students  with  an  opportunity  to  practice  the 
skills  that  were  taught  (37:19).  Tutorials  should  be 
organized  so  the  student  can  complete  them  in  30-60  minutes. 
If  the  student  completes  the  lesson  in  less  than  30  minutes, 
he  may  consider  the  lesson  a  waste  of  time.  If  it  takes  the 
student  more  than  60  minutes  to  complete  the  lesson,  he  may 
become  bored  and  disinterested  (26:5). 

The  following  list  illustrates  how  a  tutorial  program 
is  structured  (37:19;  26:6): 

1.  The  student  is  presented  with  introductory 
material . 

2.  The  student  is  presented  with  information  needed  to 
learn  the  skill. 

3.  The  student  is  then  directed  to  perform  some  action 
related  to  the  information  preeenteu,  such  as,  answering  a 
question. 

4.  The  student's  response  is  then  Judged. 

5.  The  program  provides  the  student  with  feedback  or 
remediation  in  accordance  with  the  response. 

If,  after  the  initial  response,  the  student  chooses  to 
move  ahead  or  receive  enrichment  material,  the  above  steps 
must  be  repeated  again.  When  all  the  material  is  covered 
and  the  student  has  answered  enough  questions  correctly  to 
satisfy  the  criteria  of  the  program,  he  is  informed  of  the 
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successful  completion  of  the  lesson  and  will  be  given  the 
choice  to  review  or  terminate  the  lesson  (37:19). 

Computer-Based  Tests.  Computer-based  tests  usually 
only  present  test  questions  once  unless  the  programmer 
desires  to  provide  the  student  the  opportunity  to  review 
answers  prior  to  test  completion  (37:23).  Feedback  on 
performance  is  provided  to  the  student  only  after  the 
completion  of  the  test.  However,  if  the  test  is  being  used 
as  a  pre-test  instrument,  feedback  can  be  provided  after 
each  response  (37:23;  27). 

The  following  are  the  steps  a  CBT  test  must  follow  to 
accurately  determine  student  comprehension  (37:23-24): 

1.  The  student  is  presented  introductory  material 
(directions  for  taking  the  test). 

2.  A  question  is  asked  and  the  student  provides  a 
response . 

3.  The  response  is  evaluated. 

4.  A  practice-test  program  will  then  provide  feedback. 
If  the  test  is  being  used  exclusively  for  assessment,  the 
next  question  will  be  asked. 

5.  After  all  of  the  questions  have  been  completed,  the 
student  is  provided  with  feedback. 

Computer-based  tests  can  accurately  determine 
comprehension  of  material  only  if  questions  are  formulated 
with  the  correct  content,  and  presented  in  a  clear  and 
logical  manner  (27). 
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Simulation.  Like  tutorials,  simulations  can  teach 


basic  concepts  and  principles.  However,  simulations  go  a 
step  further  into  the  teaching  of  problem  solving  (37:20). 
Simulation  programs  provide  the  student  an  opportunity  to 
experiment  with  a  realistic  situation  when  it  would  normally 
be  too  expensive  or  dangerous  to  do  so  (37:20). 

The  following  steps  are  involved  in  a  simulation 
program: 

1.  The  student  is  presented  with  concise  introductory 
material  in  order  to  correctly  operate  the  program. 

2.  The  phenomenon  (person,  object,  or  situation) 

is  pictorial ly  or  descriptively  presented  to  the  student. 

3.  The  student  takes  an  action  concerning  the 
phenomenon . 

4.  As  a  result  of  the  student's  action,  the  system  in 
which  the  phenomenon  exists  is  changed. 

5.  These  changes  in  the  system  bring  about  changes  in 
the  phenomenon,  and  are  illustrated  to  the  student 
graphically  or  descriptively. 

The  above  steps  are  repeated  until  the  student  decides 
to  terminate  the  simulation  or  the  simulation  has  been 
completed  successfully  or  unsuccessfully.  When  the 
simulation  has  been  completed,  the  student  receives  final 
results  of  his  actions  (37:21).  The  success  of  a  simulation 
program  primarily  depends  on  how  well  the  simulation 
represents  the  actual  phenomenon  (37:21). 
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Advantages  and  Disadvantages  q. f  CBT 


In  order  to  determine  if  CBT  can  be  used  in  a  training 
situation,  one  should  consider  the  advantages  and 
disadvantages  of  CBT.  The  following  lists  of  some 
advantages  and  disadvantages  of  CBT  provide  a  compilation  of 
the  opinions  of  noted  academics  and  CBT  experts. 

Advantages. 

1.  The  CBT  process  is  interactive.  It  provides  the 
student  with  an  interactive  role  in  the  training  process 
(7:85). 

2.  CBT  is  flexible  and  consistent,  allowing  trainees 
to  learn  at  their  own  pace  (7:85). 

3.  The  trainee  receives  immediate  feedback  on  his 
performance  during  the  actual  training  process  (7:85). 

4.  CBT  can  provide  the  trainee  with  a  dynamic  display 
of  material  through  the  use  of  computer  graphics  (31:216). 

5.  CBT  can  provide  training  for  large  numbers  of 
students  who  need  frequent  training,  therefore  reducing  the 
need  for  a  trainer  to  repeatedly  present  the  same  material 
(31:216). 

6.  CBT  can  be  used  to  supplement  limited  instructor 
resources  (70:20). 

7.  CBT  can  provide  a  reduction  in  the  total  time 
needed  for  training  (up  to  30  percent),  which  can  provide  a 
complementary  reduction  in  training  costs  (7:85). 
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Disadvantages. 

1.  The  amount  of  contact  a  student  has  with  people  is 
reduced  by  the  amount  of  time  he  spends  using  the  computer. 
Therefore,  personal  interaction  skills  may  suffer  (2:2). 

2.  Some  people  may  learn  better  in  other  training 
environments,  such  as  classroom  instruction  or  self-study 
(7:85). 

3.  CBT  is  of  little  value  when  the  training  goals  must 
be  changed  frequently  (31:26). 

4.  CBT  software  may  be  unre liable  (7:35). 

5.  CBT  is  a  poor  training  medium  for  presenting  large 
amounts  of  text  (31:26). 

6.  The  initial  costs  of  implementing  CBT  can  be  very 
high.  Computers  and  software  must  be  purchased  before  any 
training  can  take  place  (42:22). 

7.  Limited  amounts  of  computer  equipment  (hardware) 
may  lead  to  scheduling  problems  (7:85). 

Obviously,  all  of  the  advantages  and  disadvantages  will 
not  apply  to  every  organization.  For  example,  an 
organization  which  already  has  a  vast  supply  of  computer 
equipment  need  not  be  concerned  with  the  expense  of  the 
resources.  However,  it  is  important  to  understand  that  the 
decision  to  use  CBT  must  be  made  by  analyzing  the  advantages 
and  disadvantages  which  may  apply  to  an  organization. 
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£21  Experiments 


Three  examples  of  experiments  which  test  the 
effectiveness  of  CBT  in  training  situations  are  presented  in 
this  section  of  the  literature  review.  Each  example  is 
drawn  from  a  separate  environment  (academic,  industrial,  and 
public)  to  illustrate  how  CBT  can  be  successful  in  various 
environments . 

Academic .  A  study  was  conducted  at  Worcester 
Polytechnic  Institute,  Worcester,  Massachusetts  in  1986  to 
evaluate  the  effectiveness  of  a  CBT  program  called  PasLab. 
PasLab  is  a  CBT  program  for  instruction  in  an  initial  Pascal 
computer  language  course  (28:61). 

Samples  of  the  students  enrolled  in  the  initial  PaBLab 
course  during  terms  1984  A  and  B,  received  instruction  in 
the  traditional  Pascal  course.  Students  in  terms  1985  A  and 
B  received  instruction  utilizing  PasLab  (28:62). 

The  percentage  of  students  able  to  pass  the  Pascal 
course  in  1984  A  and  B  was  76  percent,  while  89  percent  of 
the  students  in  1985  A  and  B  who  used  PasLab  passed  the 
introductory  course.  This  study  showed  that  not  only  did 
the  use  of  PasLab  increase  the  number  of  students  who  passed 
the  Pascal  course,  it  also  encouraged  students  to  enroll  in 
more  advanced  computer  language  courses  (28:63). 

Industrial .  A  CBT  program  was  developed  for  asceptic- 
packaging  filler  operation  (for  small  drink  boxes)  training 
at  Houston  based  Coca-Cola  Foods.  The  program  consisted  of 
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seven,  1  hour  long,  training  modules  using  graphics  and 
picture  designation  design.  The  study  measured  the 
performance  of  employees  who  received  the  CBT  program  from 
October  1985  to  February  1986.  Data  was  gathered  from  four 
U.S.  plants;  three  using  CBT  and  the  fourth  serving  as  a 
control  group  (20:41). 

The  study  revealed  that  the  CBT  had  a  significant 
positive  effect  on  operating  efficiency  (1.3  percent).  The 
study  also  showed  that  the  plant's  could  expect  a  minimum  of 
$530,000  to  a  maximum  of  $1,130,000  in  net  savings  over  the 
next  five  years  due  to  the  possible  increase  in  efficiency 
as  a  result  of  training  with  the  CBT  program  (20:44). 

This  study  shows  it  is  possible  to  successfully  implement 
CBT  for  the  training  of  employees  in  an  industrial 
situation. 

Public  Sector.  A  study  conducted  by  the  Federal  Bureau 
of  Investigation  (FBI)  compared  computer-based  instruction 
to  a  highly  interactive  lecture.  The  subjects  of  the  study 
were  64  new  trainees  at  the  FBI  Academy.  The  first  group  of 
subjects  were  taught  a  block  of  training  using  a  highly 
interactive  lecture.  The  second  group  received  the  same 
block  of  training  through  computer-based  instruction.  A 
third  group  received  the  same  training  block  using  lecture 
and  computer-based  instruction.  The  test  instrument  used 
was  a  six  item  multiple  choice  test  authored  by  experts  in 
the  field  of  investigation  (64:2). 
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The  following  are  the  results  of  the  experiment: 

The  results  support  that  computer-based  instruction 
can  be  more  effective  in  fostering  student  learning. 
Students  exposed  to  the  subject  matter  only  through 
CBI  scored  significantly  higher  on  a  knowledge  test 
than  did  students  who  were  exposed  to  the  same  subject 
matter  through  only  lecture.  As  one  might  expect, 
students  who  were  exposed  to  the  subject  matter  through 
both  instruction  methods  also  scored  significantly 
higher  than  did  those  who  only  attended  the  lecture. 
However,  the  Lecture/CBT  group  did  not  score  signi¬ 
ficantly  higher  than  their  CBI  counterparts.  (64:8) 

The  results  of  this  study  show  that  computer-based 

instruction  is  a  powerful  tool  to  teach  the  process  of 

decision  making  skills  required  for  Special  Agents  in  the 

FBI  (64:10). 

Status  of  CBT  Development  in  the  Air  Force 

The  Air  Force  Supply  community  began  developing  CBT  in 
1988.  This  drive  for  CBT  development  was  spawned  by  a 
desperate  need  to  provide  some  type  of  standard  training  to 
DDA  supply  warehousemen  (54).  This  section  describes  the 
current  status  of  CBT  development  in  the  supply  career 
field,  and  presents  some  thoughts  on  the  future  of  CBT  from 
a  strategic  perspective. 

Current  Status.  With  USAF/LEYS  as  the  functional  OPR, 
the  AFLMC  and  the  3440th  TTC  are  working  together  to  design, 
develop,  and  distribute  CBT  for  use  within  the  Air  Force 
supply  community.  Their  individual  and  joint 
responsibilities  are  defined  in  a  memorandum  of  agreement 
(MOA)  (55). 
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The  bulk  of  their  efforts  has  been  geared  towards 
developing  training  modules  for  DDA  supply  warehousemen, 

AFSC  645X1.  As  of  this  writing,  the  3440th  TTC  has  20 
lessons  nearing  completion,  which  are  scheduled  for  release 
before  the  end  of  calendar  1990.  These  lessons  include 
general  information  covering  the  Air  Force  Supply  System, 
the  Materiel,  Storage  and  Distribution  Branch,  and  basic 
principles  of  warehousing  (57).  USAF/LEYS  also  provided  a 
list  of  subjects  which  they  think  should  be  developed  using 
CBT  (38).  Figure  1  provides  a  list  of  these  topics. 

The  3440th  TTC  is  authorized  12  temporary  manning 
positions  which  were  originally  scheduled  to  expire  or.  1 
July  1990,  but  were  extended  until  1  July  1991  (41). 

Training  the  personnel  who  work  in  the  CBT  development 
office  presents  a  multi-faceted  problem  because  these  people 
perform  two  of  the  three  essential  functions  of  a  good  CBT 
development  team;  program  developers  and  subject  matter 
experts  (57). 

To  become  proficient  CBT  developers,  they  must  first 
learn  the  technical  skills  required  to  use  the  authoring 
system.  Once  they  learn  how  to  use  the  features  of  the 
authoring  system,  they  must  then  learn  the  fundamentals  of 
defining  training  objectives,  identifying  important  material 
to  include  in  the  lessons,  and  presenting  information 
according  to  accepted  learning  theory.  All  of  these  steps 
are  critical  to  the  development  of  quality  CBT  (55). 
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Supply  Customer  Training 
Supply  Inspector  Course 
Funds  Management 
Research  Process 
Systems  Analyst  (5  level) 
Base  Civil  Eng.  Logistics 
Material  Control  (Generic) 

MI CAP  Management 
War  Readiness  Spares  Mgt. 
Mobility  Program  Management 
Base  Supply  Trainer 
Combat  Supply  System 
Stock  Control  Functions 
Base  Contracting  Interface 
TAR  Program 

Supply  Analysis  (64570) 
Repair  Cycle  Management 
Supply  Executive  Course 
Health  Hazard  Material  Mgt. 
Reports  and  Listings 
Equipment  Management  (AFEMS) 
Ancillary  Training  (supply) 
Supply  Weapons  Management 
Delinquent  Document  Mgt. 
Reject  Program  Management 
Gomputer/Micro  Operation 
Data  Trans  (SIFS,BLMS,etc. ) 
Post-post  Procedures  &  Ops. 
Inquiry  Selection 
How  to  Use  AFM  67-1 
Stock  Number  Characteristics 
CDC's  (645XX) 

Precious  Metals  Recovery 


Ancillary 

Proficiency 

Augment  resident 
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Figure  1.  Potential  Topics  for  Supply  Related  CBT 
(38) 


In  addition  to  the  need  for  program  developers  and 
subject  matter  experts,  a  knowledge  engineer  comprises  the 
third  important  ingredient  of  a  good  CBT  development  team. 

A  knowledge  engineer  is  defined  as  someone  who  has  an 
understanding  of  learning  theory,  and  who  can  integrate  the 
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learning  concepts  with  the  requisite  material  during  the  CBT 
development  process.  In  this  capacity,  the  knowledge 
engineer  must  be  able  to  verbalize  and  make  understandable, 
the  learning  concepts  and  theories  which  apply  to  the 
training  situation  (55).  Appendix  A  contains  a  copy  of  an 
article  written  by  the  current  supply  career  field  knowledge 
engineer.  This  article  thoroughly  describes  the  applicable 
learning  theory  for  the  training  of  645Xls,  and  further 
explains  how  this  learning  theory  should  be  integrated  into 
the  actual  design  of  the  CBT. 

According  to  the  existing  MOA,  Lt  Col  Peterson,  AFLMC, 
is  the  approved  knowledge  engineer  for  the  Air  Force  Supply 
CBT  development  program.  This  MOA  gives  him  the  authority 
to  accept,  reject,  or  change  any  of  the  CBT  modules  before 
release.  As  a  result,  no  CBT  modules  are  released  to  the 
field  without  his  consent. 

The  knowledge  engineer  has  been  a  source  of  information 
and  a  "sounding  board"  for  the  CBT  developers  since  the 
inception  of  the  CBT  development  program.  He  is  responsible 
for  giving  the  developers  a  basic  foundation  in  learning 
theory,  which  will  help  them  to  develop  quality  CBT.  Also, 
he  reviews  in-work  CBT  modules  to  identify  shortfalls, 
suggest  changes,  and  increase  the  proficiency  level  of  the 
CBT  developers  (55). 

Future  Plans .  There  is  not  an  established  plan  or 
policy  which  sets  the  course  for  the  future  direction  of  CBT 
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development  in  the  supply  career  field,  or  in  the  Air  Force 

for  that  matter.  In  an  effort  to  determine  the  future  of 

CBT,  some  of  the  individuals  who  are  involved  in  the 

development  of  CBT  at  a  strategic  level  were  interviewed. 

In  a  Nov  89  message  to  HQ  ATC,  USAF/LEYS  stated: 

We  see  CBT  as  having  a  great  potential  to  effect 
productivity  gains  in  the  areas  of  supply  (AFR  50-10) 
customer  training  (ancillary),  proficiency  training, 
and  complementary  or  supplementary  technical 
training. . .We  believe  this  effort  is  just  the  beginning 
of  innovative  ways  to  improve  training  methods.  (38) 

USAF/LEYS  stated  that  converting  supply  customer 

training  to  CBT  will  save  approximately  200,000  man  hours 

per  year,  which  customers  now  spend  in  supply  classrooms. 

In  addition  to  saving  supply  customers  time,  USAF/LEYS  sees 

many  other  benefits  which  CBT  could  provide.  Some  of  these 

are  ( 38  ) : 

1.  Manpower  savings  in  supply  training  sections 

2.  Reduce  or  alleviate  training  shortfalls 

3.  Reduce  enroute  training  requirements  or  denials  for 
personnel  going  to  short  tour  areas  or  on  short  notice 
assignments 

4.  Reduce  and  augment  mobile  training  teams 

5.  Replace  previously  cancelled  resident  courses 

6.  Complement,  supplement,  and  augment  current 
resident  courses 

7.  Replace  low  density  (attendance)  resident  courses 
USAF/LEYS  fully  supports  the  development  of  a  permanent 

CBT  development  branch  as  a  part  of  the  3440th  TTC.  They 
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believe  this  branch  should  consist  of  1  supply  CMSgt,  AFSC 
64500  (CBT  development  manager),  12  supply  subject  matter 
experts  (CBT  developers/maintainers ) ,  and  2  personnel  from 
AFSC  70/75XXX  (duplication  and  distribution  of  CBT 
software).  The  12  subject  matter  experts  should  consist  of 
7  personnel  in  AFSC  645X0,  2  personnel  in  AFSC  645X1,  and  3 
personnel  in  AFSC  645X2.  Further,  the  lowest  rank  for  any 
of  these  12  people  should  be  SSgt  (38). 

According  to  the  Assistant  Division  Chief  for  Training 
and  Systems  Development,  HQ  ATC,  ATC  acknowledges  that  in  an 
era  of  diminishing  resources,  there  is  a  need  to  develop  and 
use  new  training  technologies  such  as  CBT.  Although  ATC  has 
not  provided  any  policy  or  guidance  at  this  time,  the 
command  is  in  the  process  of  identifying  concepts  and 
principles  which  it  will  include  in  future  policy.  ATC  does 
not  expect  CBT  to  replace  existing  technical  training 
instructors,  but  rather,  to  enhance  existing  training 
courses  by  including  blocks  of  CBT  when  and  where 
appropriate  (9). 

According  to  the  Supply  Training  Staff  Officer,  HQ  ATC, 
ATC  may  offer  to  relocate  six  of  their  own  permanent  manning 
positions  to  the  CBT  development  office.  In  turn,  the  six 
remaining  positions  in  this  office  would  be  filled  using 
permanent  manning  positions  gained  from  the  major  commands. 
However,  no  official  proposal  has  been  made  (41). 
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Lt  Col  Peterson,  the  assigned  "knowledge  engineer"  for 
supply-related  CBT,  is  an  active  proponent  of  CBT.  He 
believes  that  most  3  level  technical  training  courses  could 
be  taught  using  CBT  because  courses  such  as  these  tend  to 
teach  declarative  knowledge,  not  higher  level  knowledge.  CBT 
could  potentially  replace  the  majority  of  3  skill  level 
technical  training  courses,  except  for  those  courses  which 
rely  on  interpersonal  interaction,  and  practice  on 
specialized  equipment  or  simulated  situations  to  develop 
important  job  skills  (55). 

Further,  Lt  Col  Peterson  states  that  the  demand  for  CBT 
is  increasing,  and  will  continue  to  increase  as  the  supply 
CBT  modules  are  released  to  the  field.  As  the  demand  for 
CBT  increases,  he  predicts  that  the  CBT  development  office, 
at  the  3440th  TTC,  will  not  be  able  to  develop  new  modules 
and  simultaneously  maintain  and  update  the  modules  which 
will  be  in  the  field.  He  believes  that  the  supply  community 
is  at  an  impasse  right  now  because  there  is  a  need  for  what 
CBT  can  provide,  yet  the  infrastructure  for  handling  the 
large  scale  demand  for  CBT  does  not  exist.  He  says,  "The 
Air  Force  should  decentralize  the  development  of  CBT  by 
having  CBT  development  teams  at  each  ATC  base"  (55).  He 
states  that  each  team  should  consist  of  a  commanding  officer 
or  NCO,  a  knowledge  engineer,  one  subject  matter  expert  per 
major  career  field,  a  number  of  CBT  developers,  and  an 
administrative  assistant  (55). 
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The  Need  for  Policy  and  Guidance .  Although  the 
development  and  use  of  CBT  in  the  Air  Force  is  only  in  its 
infancy,  there  has  been  an  expressed  need  for  policy  in  this 
area.  However,  it  does  not  appear  as  if  anything  firm  is 
soon  forthcoming  (9;  41;  55).  While  many  people  have 
expressed  their  vision  of  what  the  future  holds  for  the 
development  of  CBT  in  the  Air  Force,  there  is  a  more 
immediate  problem;  at  least  concerning  the  development  of 
CBT  within  the  supply  community.  As  of  July  1991,  the  CBT 
development  office  at  the  3440th  TTC  will  no  longer  exist, 
as  their  manning  positions  are  all  temporary  (41). 

Lt  Col  Peterson  says  that  it  is  not  very  likely  that 
the  Air  Staff  or  ATC  will  create  and  distribute  a  policy, 
before  1991,  which  determines  how  CBT  development  in  the  Air 
Force. will  be  organized,  funded,  or  manned.  It  is  possible 
that  an  additional  MOA  could  be  drawn  up  between  ATC  and 
AFLMC  to  allow  the  development  of  CBT  in  the  supply  career 
field  to  continue  after  July  1991,  and  the  temporary  manning 
positions  could  be  extended  indefinitely.  This  may  suffice 
as  a  band-aid  solution  for  a  much  larger  problem,  but  should 
not  be  considered  an  acceptable  long-term  solution  (55). 

Learning 

Learning  refers  to  "the  acquisition  of  knowledge  which 
causes  a  relatively  permanent  change  in  specific 
behaviors ...  and  is  more  than  jus^  remembering”  (56:2).  It 
is  the  ability  to  perform  some  task  with  skill. 
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In  the  Air  Force,  the  traditional  technical  training 
(learning)  methods  are  based  on  “the  behavioralist 
traditions  of  B.F.  Skinner ... (who  stated)  that  human 
behaviors  are  controlled  by  the  immediate  consequences 
following  a  behavior"  (56:2).  As  a  result,  positively 
reinforced  behaviors  tend  to  be  repeated.  Applying 
Skinner's  operant  conditioning  principles  to  programmed 
instruction  have  reduced  the  amount  of  time  required  to 
teach  a  technical  skill  (56:2). 

Learning  can  be  described  in  many  ways.  In  the  context 
of  this  research,  we  will  consider  2  established  scientific 
viewpoints;  structural  and  functional.  Also,  we  will  look 
at  a  third  viewpoint  called  mastery  learning.  A  structural 
viewpoint,  which  is  often  associated  with  cognitive 
psychology,  emphasizes  "what"  and  "when"  learning  occurs. 

The  functional  perspective,  on  the  other  hand,  describes 
under  what  "conditions"  learning  occurs  (17:32). 

Structural  Perspective .  The  structural  perspective 
defines  learning  in  terms  of  human  information  processing. 
Information  from  the  environment  is  input  into  the  sensory 
register  of  the  brain.  Mental  activities  in  the  brain 
include  pattern  recognition,  short-term  memory,  and  long¬ 
term  memory.  Input  is  encoded  in  the  brain  in  short-term 
and  long-term  memory  (18:32).  Most  information  processing 
models  begin  with  the  need  for  the  learner  to  attend  to  the 
instructional  material.  Once  the  learner  is  attending,  the 
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information  is  coded  and  stored  temporarily  in  the  working 
memory.  Much  of  the  information  stored  in  the  working 
memory  is  lost,  but  some  of  it  is  incorporated  into  long¬ 
term  memory.  Actual  learning  occurs  when  this  final  Zink  is 
made  into  long-term  memory  (13:79). 

Cognitive  scientists  realize  the  role  of  the  trainee  as 
a  passive  receiver  of  information  has  changed  (18:91). 

Teaching  and  learning  involve  a  transaction.  As  such, 
the  process  is  an  active  one,  not  only  for  instructors, 
but  also  for  trainees.  Unfortunately,  learners  are 
often  cast  in  the  passive  role... One  of  the  key 
functions. .. is  to  identify  an  appropriate  set  of 
learning  activities  and  experiences.  In  this  way, 
trainees  can  become  active  partners  in  the  teaching¬ 
learning  process.  (18:91) 

New  cognitive  learning  theories  promote  the  concept  of 

cognitive  model  development.  This  model  consists  of 

declarative  knowledge,  procedural  knowledge,  and  systems 

knowledge.  Declarative  knowledge  refers  to  specific  tasks 

or  principles  about  the  task.  Procedural  knowledge  refers 

to  how  the  task  is  performed.  Systems  knowledge  refers  to 

how  the  task  fits  into  other  tasks  (58:4).  There  is  a 

tendency  for  cognitive  models  to  increase  the  complexity  of 

training  technical  tasks. 

To  institute  this  form  of  training  in  a  classroom 
environment  would  mean  substantially  extending  the 
length  of  most  resident  courses.  The  desire  of  most 
organizations  to  reduce  classroom  time  and  move 
training  closer  to  the  actual  work  environment,  makes 
the  use  of  cognitive  learning  theories  very  unlikely. 
(56:3) 
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However,  it  may  be  possible  to  improve  learning  and 
reduce  training  time  by  combining  the  new  cognitive  learning 
theories  with  CBT  (56:4). 

Functional  Perspective .  The  thesis  of  this  concept 
states  that  the  relationship  between  the  environment  and 
what  a  person  does,  has  profound  implications  on  learning. 
The  present  environment  is  not  the  only  thing  that  helps  or 
hinders  learning.  Changing  events  which  occur  before  or 
after  training  will  also  effect  the  students'  behavior 
(17:40) . 

Functional  psychologists  state  that  learning  is  not 
determined  by  age  or  personality,  but  by  "species 
characteristics,  biological  maturation,  or  the  history  of 
interaction  with  a  particular  environment  from  the  moment  of 
conception"  (17:33).  In  terms  of  CBT,  the  functional 
perspective  indicates  that  students  will  learn  depending  on 
how  well  they  have  learned  in  the  past  (17:43). 

Mastery  Learning.  Mastery  learning  is  achieved  when  a 
person  becomes  so  familiar  with  a  subject  matter,  that  he 
can  perform  with  accuracy  and  speed.  For  example,  when 
someone  achieves  mastery  in  a  cognitive  skill  such  as 
learning  a  foreign  language,  he  is  able  to  speak  the  new 
language  fluently.  "In  situations  where  mastery  of  the 
subject  matter  is  necessary,  it  is  often  the  technique  of 
choice"  (16). 
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Mastery  learning  is  appropriate  for  situations  where  a 
person  needs  to  know  information  to  be  able  to  do  a  job 
safely,  efficiently,  or  effectively.  It  is  most  applicable 
when  the  content  of  the  knowledge  can  be  expressed  in  verbal 
form,  and  when  it  is  conceptual  and  factual  in  nature  (as 
opposed  to  physical).  Also,  it  is  appropriate  for  learning 
information  "on  licensing  tests,  product  knowledge, 
operating  procedures,  concept  information,  computer  skills, 
and  academic  skills"  (16). 

Cowardin  says  that  a  behavioral  learning  system  best 
complements  the  mastery  learning  concept.  The  behavioral 
approach  defines  learning  as  a  change  in  behavior  "which 
emerges  during  the  lifetime  of  an  individual",  and  is 
measured  best  by  frequency  of  response.  Behavioral  learning 
involves  repeated  measurement  of  behavior  over  time.  This 
approach  does  not  "just  present  the  material  and  leave  the 
learning  up  to  the  student".  Instead,  this  concept  is 
interested  in  engineering  behavioral  change  toward  a 
reasonable  goal.  The  7  key  features  of  a  behavioral 
learning  system  are  (16): 

1.  Define  the  goal  in  measurable  terms. 

2.  Rely  on  the  rate  of  response  as  a  salient  measure. 

3.  Control  the  consequences  of  relevant  behaviors. 

4.  Evaluate  methods  as  well  as  the  students' 
performance,  especially  if  the  performance  goals  are  not 
met . 
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5.  Assume  the  student  can  learn  from  a  properly 
controlled  environment. 

6.  Measure  often. 

7.  Employ  a  measurement  that  has  scientific  rigor. 

Important  Principles  q!  Learning.  There  are  many 

tactics  and  strategies  which  can  be  used  to  promote 
learning.  Davies  provides  the  following  general  principles, 
which  may  act  as  guidelines  to  facilitate  learning  (18:249): 

1.  Learners  must  be  active  rather  than  passive. 

2.  Frequency  and  recency  are  important.  The  more 
something  is  done,  the  more  recent  the  performance,  the 
better  the  learning. 

3.  Learning  is  purposeful;  it  is  not  reflexive.  There 
is  a  conscious  purpose,  and  this  motivation  needs  to  be 
recognized. 

4.  Reinforcement  and  feedback  are  essential.  If 
people  are  not  rewarded  they  will  not  learn.  If  they  don't 
get  feedback,  their  learning  will  not  improve.  They  will 
fall  into  bad  habits. 

5.  Learning  should  be  pleasant  and  enjoyable. 
Unpleasant  situations  are  avoided. 

Measuring  Learning.  Often,  learning  is  measured  using 
a  test  or  examination,  which  can  be  written  or  oral, 
knowledge  or  skills  based.  However,  these  learning  measures 
are  indirect  and  may  not  reflect  actual  learning.  On  the 
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other  hand,  good  test  scores  may  exaggerate  learning,  as 
when  a  trainee  guesses  a  correct  answer  (18:229). 

It  is  possible  that  traditional  testing  is  not  the  best 
way  to  measure  what  has  been  learned.  In  the  Air  Force, 
this  is  evident  by  the  inability  of  many  technical  school 
graduates  (who  pass  the  end-of-the-course  test)  who  reach 
their  first  operational  assignment,  only  to  find  that  they 
cannot  successfully  perform  the  simple  tasks  for  which  they 
were  trained  (55).  Despite  its  problems,  measuring  learning 
does  have  a  place  in  assessment.  As  long  as  care  is  taken 
in  the  design  of  tests  and  examinations,  valuable 
information  can  be  obtained  (18:229). 

Tests  can  fall  into  two  categories,  descriptive  or 
prescriptive.  Descriptive  tests  which  are  analogous  to 
traditional  written  examinat ions ,  measure  the  students 
ability  to  remember  information.  On  the  other  hand, 
prescriptive  tests  measure  the  students  ability  to  integrate 
the  learned  knowledge,  in  order  to  accomplish  a  task 
requiring  a  series  of  steps.  Performance  testing  is  nothing 
more  than  measuring  how  well  the  student  performs  in  an 
actual  or  simulated  work  environment  (55).  Many  experts 
agree  that  for  many  technical  training  applications, 
learning  is  best  measured  by  prescriptive  testing 
(18:229;  56). 
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Training 


In  a  broad  sense,  the  concept  of  training  means  "doing 
something  to  others  so  that  they  will  do  something 
differently"  (43:48).  The  goal  of  technical,  or  skills 
training  programs  should  be  to  give  the  learner  the  ability 
to  reproduce  a  skill  with  accuracy,  and  to  combine  more  than 
one  skill  to  complete  a  job  (66:39).  Many  organizations 
recognize  that  training  is  an  important  variable  which 
affects  productivity  (12:64;  53:47).  However,  training 
programs  within  organizations  are  often  "ad  hoc  and  reactive 
functions"  (53:49). 

The  term,  "positive  transfer  of  training",  is  often 
used  to  measure  the  degree  to  which  trainees  can  effectively 
apply  the  knowledge,  skills,  and  attitudes  gained  in  a 
training  context  to  the  job  (8:63).  Baldwin  and  Ford 
proposed  a  model  which  compares  the  relationship  between 
training  inputs,  training  outputs,  and  conditions  of 
transfer  (8:63).  Baldwin  and  Ford  state  that  training 
inputs  include  the  trainees'  personal  characteristics,  the 
training  design,  and  the  work  environment,  while  the 
training  outputs  include  learning  and  retention.  Conditions 
of  transfer  consider  such  concepts  as  maintenance  and  the 
ability  to  generalize.  Baldwin  and  Ford  concluded  that  for 
positive  transfer  to  occur,  the  learned  behavior  must  be 
generalized  to  the  job  context  and  maintained  over  a  period 
of  time  while  on  the  job  (8:64). 
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Four  principles  of  training  have  been  identified  which 
help  to  improve  the  positive  transfer  of  training.  They  are 
as  follows  (12:65): 

1.  Make  training  content  relevant  to  the  trainees. 

2.  Make  training  objectives  congruent  with  the  work 
tasks  you  are  trying  to  affect. 

3.  Make  the  training  design  systematic  and  be  sure  it 
relates  directly  to  the  training  objectives. 

4.  Plan  training  delivery  to  respond  to  the  trainees 
frame  of  reference. 

When  a  training  program  proves  to  be  ineffective,  few 

will  consider  the  idea  that  the  wrong  training  tool  was 

chosen.  Instead,  "the  belief  that  training  is  not  really 

important  or  effective  tends  to  be  reinforced"  (40:26). 

Too  often,  when  an  organization  is  looking  into 
using  new  technologies,  the  trainers  will  get  hung 
up  on  the  methodology  and  forget  these  concepts 
behind  the  process.  The  key  is  the  job  to  be  done 
should  dictate  the  tools  used,  not  the  other  way 
around.  (40:26) 

To  ensure  that  training  is  properly  planned,  organized,  and 
delivered,  an  8  step  model  for  developing  a  training 
strategy  was  proposed.  This  model  highlights  some  of  the 
important  areas  which  must  be  considered  throughout  the 
training  development  process.  These  8  steps  are  (40:25): 

1.  Find  out  what  the  problem  is,  and  what  learning  is 
needed  to  solve  it. 

2.  Analyze  your  training  audience. 

3.  Develop  specific,  measurable  learning  objectives. 
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4.  Decide  which  skills  and  attitudes  will  help  the 
trainees  reach  their  goal. 

5.  Examine  the  organizational  resources  and 
constraints . 

6.  Design  the  training  program. 

7.  Produce  and  implement  the  program. 

8.  Evaluate  the  program  to  determine  its  effective¬ 
ness. 

Diversity  of  Training.  The  diversity  of  training  tools 
and  the  techniques  which  are  available  seems  almost 
limitless.  Given  the  apparent  overabundance  of 
possibilities,  how  does  an  organization  answer  the  question, 
what  is  the  best  training  media  to  use?  "The  'perfectly 
professional'  answer  should  be  a  resounding  'it  depends'” 
(39:96).  The  material,  audience,  location,  and  objectives 
are  important  factors  which  organizations  should  consider 
when  choosing  a  methodology  (45:57).  Figure  2  provides  a 
list  of  some  training  techniques  and  tools  which  are 
commonly  used  today. 

The  Learning  Continuum.  Ganger  and  Christensen  define 
the  Learning  Continuum  as  "a  continuous  set  of  learning 
activities  which  allows  the  user  to  acquire  knowledge  about 
his  or  her  Job"  (25).  They  believe  that  the  proliferation 
of  microcomputers  in  the  workplace  has  empowered  employees, 
allowing  them  to  generate  and  receive  more  information  and 
data  than  ever  before.  "Empowerment  has  made  the  average 
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job  increasingly  complex  and  demanding,  and  more  dependent 
on  cognitive  and  decision  making-skills"  (25). 


Techniques 

Tools 

Artificial  Intelligence 

Audiocassettes 

Business  Gaming 

Flipcharts 

Case  Studies 

Movies 

CBT 

Ovrhd.  Projectors 

Concurrent  Training 

Sound  Filmstrips 

Demonst r at ion 

Slides 

Electronic  Workbooks 

Textbooks 

Interactive  Videodisk 
Lecture 

One-to-One  Instruction 
Problem  Solving 
Role-Playing 

Videocassettes 

Satellite  Delivered  Learning 

Self  Instruction 

Seminar  Style  Training 

Simulations 

Teleconferencing 

V ideoconf erenc ing 

Figure  2.  Training  Techniques  and  Tools  (5:91; 

14:31;  35:32;  40:26;  45:57;  65:30;  71:84) 


The  Learning  Continuum  is  a  training  approach  which 
will  help  ensure  success  for  employees  in  the  1990s.  This 
concept  states  that  training  is  an  "on-going  process  and, 
like  other  resources,  should  reside  at  the  individual 
workers  workstation  or  personal  computer  (PC)"  (25).  The 
worker  should  have  the  power  of  deciding  when  and  in  what 
form  the  training  will  take  place.  As  a  result,  training 
will  integrate  the  new  skills  into  the  context  of  the 
workers  job. 
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They  propose  that  the  Learning  Continuum  is  very  well 
suited  for  a  CBT  environment.  Figure  3  identifies  some 
adult  learning  needs  which  the  Learning  Continuum  can 
address  through  a  CBT  medium. 


Adult  Learning  Needs 

The  Learning  Continuum 

1 .  Adults  need  to 
know  why 

1. 

Pre-training  activities 
introduce  change,  and 
remove  fear  of  the  un¬ 
known 

2.  Adults  need  to 
have  control 

2. 

Training  occurs  on  demand 
and  is  available  as 
required 

3.  Adults  need  to 
have  the  learning 
experience  linked 
to  the  job 

3. 

Activities  are  directly 
related  to  the  job 

4. 'Adults  need  to  4. 

learn  at  their  own 
pace,  and  to  have 
sufficient  time  to 
master  tasks  and  skills 

Learning  is  on-going  and 
learners  work  at  their 
own  pace 

Figure  3.  Comparison  of  Adult  Learning  Needs  to 

Capabilities  of  the  Learning  Continuum  (25) 


The  Learning  Continuum  consists  of  three  levels; 
Preparation  for  the  acquisition  of  knowledge.  Acquisition  of 
knowledge  and  skills,  and  Retention  of  knowledge  and 
sharpening  of  skills.  Figure  4  provides  a  description  of 
these  3  phases  in  more  detail. 
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1.  Pre-class  work:  Provides  the  necessary  in¬ 
formation  to  establish  the  baseline  level  of 
knowledge  and  understanding  required. 

B.  Acquisition  of  Knowledge  and  Skills 

1.  Tutorials:  Prevent  the  training  gap,  solves 
the  travel  problem,  and  accommodates  adult 
learning  styles. 

2.  Practice  opportunities  and  simulations: 
Prevents  errors  and  supports  classroom 
instruction  by  providing  an  opportunity  to 
try  new  skills  before  using  them  in  a  real 
situation. 

C.  Retention  q±  Knowledge  and  Sharpening  q±  Skills 

1.  Practice  and  simulation:  Skills  need  to  be 
practiced  and  learning  needs  to  be  reinforced. 
Especially  true  when  skills  are  used  often. 

2.  Testing:  The  use  of  testing  programs  allows 
users  to  check  their  own  skill  levels  and  spot 
weak  areas. 

3.  Performance  support  tools:  These  provide 
"just-in-time  training",  providing  the 
information  when  it  is  needed. 


Figure  4.  Three  Phases  of  the  Learning  Continuum  (25) 


QUEST  Authoring  System 

The  QUEST  Authoring  System  was  designed  to  be  used  by 

trainers,  teachers,  and  anyone  else  who  trains  and  educates 

people.  QUEST  makes  it  possible  for  the  non-programmer  to 

translate  his  thoughts  and  training  ideas  into  computer- 

based  training  lessons  and  tests  (4:v). 

QUEST  is  an  authoring  system,  not  an  authoring 
language .  The  difference  being  that  an  authoring 
language  requires  programming  skills  to  use  it.  With 
QUEST  you  can  perform  many  of  the  same  functions  using 
menu  screens  and  prompt  lines  which  display  the 
available  functions  in  plain  English.  (4:v) 
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Therefore,  people  with  no  programming  experience  can 
create  very  sophisticated  CBT  programs  utilizing,  nice , 

feedback,  complex  branching,  internal  answer  evaluation 
techniques,  and  much  more.  QUEST  is  a  very  flexible 
program.  It  allows  the  user  to  transfer  his  own  -unique 
teaching  style  into  CBT  programs  (4:v). 

QUEST  Strm  ;ture .  The  QUEST  Authoring  System  consists 
of  several  programs.  These  programs  are  interrelated  and 
are  centered  around  the  main  AUTHOR  program.  The  different 
programs  make  it  possible  for  the  user  to  create,  manage, 
and  present  CBT  lessons  (4:vi).  Figure  5  illustrates  the 
relationship  between  the  QUEST  programs. 


Figure  5.  QUEST  Configuration  (4:vi) 
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The  Computer  Managed  Instruction  (CMI)  programs  are 
programs  used  as  administrative  devices.  There  are  3  CMI 
programs:  REGISTER,  CATALOG,  and  REPORT.  The  REGISTER 
program  controls  access  to  QUEST,  only  permitting  registered 
users  to  enter  programs.  The  CATALOG  program  establishes  a 
catalog  of  the  available  lessons  students  can  take.  The 
REPORT  program  provides  the  manager  or  trainer  with  student 
lesson  reports  which  can  be  used  for  student  evaluation 
(4: vii ) . 

The  AUTHOR  program  is  the  main  program  used  to  actually 

create  the  CBT  lessons.  All  the  other  programs  within 

QUEST,  with  the  exception  of  CMI  programs,  are  additional 

tools  used  to  enhance  the  lesson  created  in  AUTHOR  (4:vi). 

AUTHOR  is  the  primary  tool  for  creating  your  CBT 
courses.  It  allows  you  to  enter  text,  draw  graphics 
and  illustrations,  create  animation,  set  up  branching, 
ask  questions  and  specify  how  the  student's  answer  will 
be  analyzed,  provide  feedback,  and  many  other  options. 
(4:vi) 

The  LEARN  program  allows  the  students  to  view  the 
lesson,  answer  questions,  take  tests,  and  receive  feedback 
on  performance  during  the  lesson  (4: vii). 

The  QUEST  Editor  programs  expand  the  power  of  the  text 
and  graphics  capabilities  found  in  the  AUTHOR  program. 

There  are  three  separate  editors  available:  Shape  Editor, 
Character  Editor,  and  the  Text  Editor.  The  Shape  Editor 
allows  the  user  to  create  graphics  and  store  them  for  later 
use  when  developing  lessons.  The  Character  Editor  allows 
the  user  to  create  unique  character  sets,  such  as 
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mathematical  symbols  and  foreign  language  characters.  The 

final  editor  available  is  the  Text  Editor.  The  Text  Editor 

is  used  to  write  messages  and  batch  files.  While  in  the 

LEARN  program,  students  can  use  the  Text  Editor  to  write 

messages  to  their  instructor  (4:vii). 

The  QUEST  Authoring  Language  (QAL)  provides  the  user 

with  the  power  to  develop  extremely  sophisticated  CBT 

lessons  which  are  beyond  the  abilities  of  the  QUEST 

Authoring  System  used  alone.  The  structure  and  logic  of  QAL 

is  very  similar  to  Pascal  (4:viii). 

A  QAL  program  could  be  used  to  transfer  information 
between  a  database  program  and  a  QUEST  lesson,  or  to 
monitor  various  gauges  and  instruments  in  a  flight 
simulation.  (4:viii) 

There  are  several  Auxiliary  programs  available  to  the 
user.  The  Auxiliary  programs  allow  the  user  to  import 
images  from  outside  programs,  check  for  flaws  in  lesson 
structure,  and  many  more  available  functions  (4:viii). 

The  Storage  and  Distribution  Entry-Level  CBT  Program 

This  CBT  program,  which  is  used  in  this  study,  was 
designed  by  the  Lowry  AFB  CBT  team  and  developed  by  the 
AFLMC  programmer  using  the  QUEST  Authoring  System.  The 
program  consists  of  three  parts;  tutorial,  drill  and 
practice,  and  a  CBT  Mastery  test. 

Tutorial .  The  tutorial  begins  with  an  introductory 
portion  which  explains  to  the  test  subjects  that  after  the 
lesson  is  complete  they  will  be  able  to  properly  store 
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property  in  the  correct  warehouse  location.  Also,  the 
introductory  portion  tells  the  students  that  they  will  need 
to  use  two  chapters  from  AFM  67-1  for  reference  during  the 
lesson.  The  lesson  portion  of  the  program  uses  the 
techniques  of  feedback  and  remediation  when  necessary.  The 
tutorial  is  organized  into  three  knowledge  clusters.  Each 
cluster  teaches  the  students  one  important  procedure  or 
technique  involved  with  the  correct  storage  of  property  in  a 
warehouse  location.  The  first  cluster  instructs  students  on 
the  difference  between  due-out-releases  and  management 
notices,  and  the  two  different  types  (1046  and  1102)  of 
management  notices  used  to  store  property.  The  second 
cluster  introduces  the  students  to  the  breakdown  of  a 
warehouse  location.  Finally,  the  third  cluster  instructs 
the  students  on  the  five  step  process  for  correctly  storing 
property  in  a  warehouse  location. 

The  tutorial  utilizes  a  combination  of  lesson 
(instructional)  screens,  memory  check  (questions)  screens, 
and  summary  screens  to  teach  the  students  to  correctly 
warehouse  property.  Lesson  screens  are  used  to  present  the 
material  to  the  students.  The  lesson  screens  were  designed 
with  a  combination  of  graphics  and  text  for  the  presentation 
of  the  instructional  material. 

After  a  few  lesson  screens  are  presented  to  the 
students,  memory  check  questions  are  used  to  determine  if 
they  comprehended  the  previously  presented  material.  Memory 
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check  questions  can  be  true/false,  multiple  choice,  f ill- 
in-  the-blank,  or  matching  questions.  If  the  student 
correctly  answers  the  memory  check  question,  positive 
feedback  will  be  displayed  on  the  memory  check  screen. 
However,  if  the  student  answers  incorrectly,  he  will  be 
given  feedback  and  may  be  diverted  to  remediation. 
Remediation  consists  of  several  lesson  screens  structured 
slightly  differently  than  the  previous  lesson  screens  on  the 
material.  After  the  remediation  lesson  screens  are 
presented,  memory  check  questions  again  check  for  student 
comprehension.  If  the  student  still  has  problems  answering 
the  questions  correctly  the  program  stops  the  lesson  and 
directs  the  student  to  contact  his  supervisor  or  trainer  for 
assistance.  The  last  screen  in  each  knowledge  cluster 
provides  a  summary  of  what  the  student  should  have  learned. 
Figure  6  shows  a  flow  chart  of  the  presentation  logic  in  the 
tutorial  portion  of  the  program.  Knowledge  cluster  3 
follows  the  same  general  pattern  as  clusters  1  and  2.  The 
numbers  in  each  box  represent  the  number  of  each  type  of 
screen  presented. 

Drill  and  Practice .  The  drill  and  practice  portion  of 
the  program  enables  the  student  to  fine  tune  the  skills  and 
knowledge  attained  from  the  tutorial.  Prior  to  beginning 
the  practice  questions  the  students  are  given  the 
opportunity  to  continue  with  the  drill  and  practice  portion 
of  the  program,  review  the  tutorial  portion,  or  quit  the 
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program  and  reenter  at  a  later  time  at  the  drill  and 
practice  location. 
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Figure  6.  Flow  Chart  of  Tutorial  in  the  CBT 
Storage  and  Distribution  Entry 
Level  Program 
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The  drill  and  practice  consists  of  eight  questions  very 
similar  to  the  memory  check  questions  in  the  tutorial.  If 
the  student  answers  the  questions  correctly,  positive 
feedback  is  displayed.  If  the  student  answers  the  question 
incorrectly,  he  will  either  receive  feedback  explaining  why 
the  answer  was  incorrect,  or  he  will  receive  feedback  along 
with  remediation  on  the  material  in  question.  The  flow 
chart  of  the  drill  and  practice  portion  of  the  program  is 
shown  in  Figure  7. 

Mastery  Test.  The  Mastery  Test  v-as  developed  to  test 
the  student's  knowledge  in  three  knowledge  clusters.  The 
Mastery  Test  contains  13  primary  and  5  adaptive  questions. 
The  adaptive  questions  are  added  when  certain  questions  are 
answered  incorrectly.  The  adaptive  questions  give  the  test 
subject  the  opportunity  to  answer  questions  which  are 
similar  to  the  one(s)  that  were  originally  missed.  The  test 
subjects  are  not  penalized  if  they  answer  these  additional 
questions  correctly.  No  feedback  is  given  to  the  test 
subjects  because  the  Mastery  Test  is  used  only  for  the 
purpose  of  evaluation.  A  flow  chart  of  the  sequence  of  the 
Mastery  Test  questions  is  depicted  in  Figure  8.  The  first 
knowledge  cluster  contains  two  Yes/No  primary  questions  on 
identifying  if  property  has  been  delivered  to  the  correct 
warehouse.  The  second  knowledge  cluster  uses  4  f ill-in- the- 
blank  questions  dealing  with  the  identification  of  the 
correct  stockroom,  bin  row,  horizontal  bin  level,  and 
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Figure  7.  Flow  Chart  of  Drill  and  Practice 
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vertical  bin  level  locations.  The  third  and  final  cluster 
contains  6  multiple  choice  questions  and  a  f ill-in-the-blank 
question  pertaining  to  the  5  step  procedure  for  storing 
property  in  the  correct  warehouse  location. 


Summary 

This  chapter  reviewed  the  current  literature  relating 
to  CBT.  Included  was  a  review  of  the  history  of  CBT, 
methodology  and  design  of  CBT,  advantages  and  disadvantages 
of  using  CBT,  and  a  discussion  of  some  prior  experiments 
which  examined  the  effectiveness  of  CBT.  Also,  the  current 
and  future  status  of  CBT  development  in  the  Air  Force  was 
discussed,  as  were  current  learning  theory  and  training 
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concepts.  Further,  the  QUEST  authoring  system  was  reviewed, 
and  a  discussion  of  the  Storage  and  Distribution  Entry-Level 
CBT  program  was  included. 

This  review  of  the  literature  provided  a  broad,  yet 
thorough,  understanding  of  many  important  topics  which  are 
directly  related  to  this  research.  Chapter  III  expands  on 
this  understanding,  by  including  a  review  of  the  literature 
as  it  pertains  to  the  methodology.  Further,  Chapter  III 
describes  the  methodology  which  was  used  to  objectively 
examine  the  research  hypothesis. 
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Ill .  Methodology 


Introduction 

This  chapter  explains  how  the  research  problem  was 
investigated  using  an  experimental  research  design.  The 
research  problem  directly  lends  itself  to  experimentation 
because,  “experiments  are  studies  whose  implementation 
involves  intervention  by  the  researcher  beyond  that  required 
for  measurement"  (23:130).  Intervention  is  required  in 
order  to  determine  the  relative  significance  of  the 
experimental  treatments.  Without  the  researchers' 
intervention,  it  would  be  difficult  to  adequately  compare 
the  experimental  treatments,  and  to  identify  any  significant 
differences  existing  between  the  treatments.  The 
experimental  methodology  will  make  it  possible  to  identify 
significant  changes  in  the  dependent  variable,  which  result 
from  manipulation  of  the  independent  variable. 

This  chapter  reviews  the  pertinent  methodological 
literature,  and  restates  the  primary  research  hypothesis. 
Also,  the  independent  and  dependent  variables  are  discussed, 
as  are  the  experimental  design  and  the  research  procedures 
used  to  conduct  this  experiment.  In  addition,  the  sampling 
method  is  discussed,  and  the  methods  of  data  analysis  are 
explained  to  provide  a  framework  for  how  the  data  will  be 
analyzed  in  Chapter  IV. 
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Literature  Review  of  the  Methodology 


This  literature  review  is  discussed  in  two  parts. 

First,  the  research  methods  and  design  used  in  this  study 
are  discussed.  This  discussion  includes  the  post-test-only 
control  group  design,  and  the  stratified  random  sampling 
technique. 

In  the  second  portion  of  this  review,  specific 
statistical  techniques  are  discussed.  This  discussion 
includes  both  parametric  and  non-par ame trie  techniques  for 
analyzing  the  data.  The  discussion  of  parametric  techniques 
will  include  the  Central  Limit  Theorem  (CLT),  the  one-way 
analysis  of  variance  (ANOVA),  the  Bonferroni  multiple 
comparison  procedure,  and  the  pooled  variance  t-test.  The 
discussion  of  non-par ame trie  techniques  includes  the 
Kruskal-Wallis  One-way  ANOVA  by  Rank  (K-W  ANOVA)  and  the 
Mann -Whitney  U  test. 

Research  Design.  The  post-test-only  control  group 
design  does  not  rely  on  the  pre-testing  of  the  test 
subjects.  Emory  believes  that  pre-tests  are  not  necessary 
when  samples  are  selected  at  random  (23:122).  He  justifies 
this  belief  by  Baying,  "internal  validity  threats  from 
history,  maturation,  selection,  and  statistical  regression 
are  adequately  controlled  by  random  assignment"  (23:122). 

The  experimental  results  of  the  post-test-only  control  group 
design  are  measured  by  taking  the  difference  between  the 
post-test  results  of  the  various  experimental  groups  (19). 
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This  methodology  measures  experimental  effect  by 

comparing  the  differences  in  post-test  scores  between  the 

various  treatment  groups.  The  post-test-only  control  group 

design  is  effective  when  samples  are  randomly  selected. 

Also,  this  design  eliminates  the  "threat  of  testing",  which 

is  the  possibility  that  pre-testing  may  give  the  test 

subjects  an  idea  of  the  expected  or  hypothesized  outcome  of 

the  experiment  (19).  Although  this  design  is  very  powerful, 

it  is  subject  to  certain  limitations.  "Differential 

mortality  between  experimental  and  control  groups  continues 

to  be  a  potential  problem"  (23:122). 

The  stratified  random  sampling  technique  partitions,  or 

stratifies,  the  population  into  a  number  of  mutually 

exclusive  subgroups,  or  strata  (23:306;  47:308).  This 

sampling  method,  "is  almost  always  more  efficient 

statistically  than  simple  random  sampling  and  at  its  worst 

is  equal  to  it"  (23:307).  Stratified  random  sampling  is 

more  powerful  because  it  ensures  that  sub-groups  are 

represented  in  the  same  proportion  in  each  of  the  three 

treatments  as  they  are  found  in  the  true  population  (19). 

Stratified  sampling  usually  provides  more  accurate 
estimates  of  the  population  mean  than  does  a  simple 
random  sample  of  the  same  size  because  the  variability 
within  the  strata  is  usually  less  than  the  variability 
over  the  entire  population.  (48:1148) 

There  is  no  standard  procedure  to  determine  how  many 

strata  to  use.  Emory  states  that,  "theoretically  the  more 

strata,  the  closer  you  are  likely  to  come  to  maximizing 
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inter-strata  differences  and  minimizing  intra-stratum 
variances"  (23:308).  Advantages  of  stratified  random 
sampling  are  as  follows  (23:307): 

1.  Increases  the  samples  statistical  efficiency. 

2.  Provides  adequate  data  for  analyzing  the  sub¬ 
populations  . 

3.  Enables  different  research  methods  and  procedures  to 
be  used  in  different  strata. 

Statistical  Methods.  This  section  describes  the 
difference  between  parametric  and  non-parametric  statistics, 
and  the  specific  parametric  and  non-parametric  techniques 
which  may  be  used  to  analyze  the  experimental  data.  The 
actual  sample  size,  distribution,  variability,  and  sampling 
method  will  determine  whether  parametric  or  non-parametric 
techniques  are  used.  When  using  parametric  statistics,  the 
data  must  meet  certain  criteria  or  assumptions.  If  the  data 
does  not  meet  these  criteria,  then  non-parametric  statistics 
will  be  used. 

Parametric.  Statistics.  Siegel  defines  a  parametric 
statistical  test  as  a  test  "whose  model  specifies  certain 
conditions  about  the  parameters  of  the  population  from  which 
the  research  sample  was  drawn"  (60:30-31).  There  is  no 
common  agreement  as  to  what  constitutes  a  sufficiently  large 
sample  size.  Unless  the  population  is  approximately  normal, 
small  samples  generally  do  not  result  in  a  sampling 
distribution  that  is  normally  distributed  (47:315).  The 
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Central  Limit  Theorem  (CLT),  "provides  the  theoretical 

framework  for  most  statistical  estimation  and  tests  of 

hypothesis"  (47:316).  The  CLT  implies  that  regardless  of 

the  shape  of  the  frequency  distribution  of  the  population, 

the  frequency  distribution  of  the  sample  mean,  in  repeated 

samples  of  a  certain  size,  tends  to  approach  a  normal 

distribution  as  the  sample  size  increases  (10:73;  62:47). 

Many  researchers  consider  that  a  sample  size  of  30  or  more 

qualifies  as  a  large  sample  (19;  33:273;  35;  47:315; 

48:321).  The  CLT  is  easy  to  apply  because  few  assumptions 

must  be  met  in  order  to  use  it. 

Apart  from  the  condition  of  random  sampling,  the 
theorem  requires  very  few  assumptions;  it  is  sufficient 
to  say  that  sigma  is  finite  and  that  the  sample  iB  a 
random  sample  from  the  population.  (62:48) 

The  one-way  ANOVA  is  used  when  the  experiment  uses  a 

single  independent  variable  to  estimate  the  dependent 

variable  (35;  49:91).  This  test  is  used  to  determine  if 

there  is  a  statistical  difference  between  the  means  of  two 

or  more  samples  (47:395).  The  one-way  ANOVA  uses  the  F-test 

statistic,  which  is  a  measure  of  the  between  sample 

variability  divided  by  the  within  sample  variability  (35; 

49:91).  The  one-way  ANOVA  is  based  on  the  following  three 

assumpt ions  (10:245): 

1.  Samples  come  from  normal  populations. 

2.  Random  and  independent  sampling  is  used. 

3.  Population  variance  is  constant  among  groups. 
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Figure  9  describes  the  one-way  ANOVA  hypothesis  test,  and 
provides  the  criteria  for  rejection  of  the  null  hypothesis 
(35). 

Although  the  one-way  ANOVA  can  determine  if  at  least 
one  of  the  sample  means  is  significantly  different,  it 
cannot  determine  which  specific  sample  means  are  different. 
There  are  several  multiple  comparison  techniques  to  choose 
from  which  will  determine  exactly  where  these  differences 
lie  (35).  This  review  will  only  consider  one  of  these 
multiple  comparison  techniques,  the  Bonferroni  procedure. 
This  procedure  results  in  confidence  intervals  that  are  at 
least  100 ( 1-alpha )%  confident,  and  is  a  function  of  the  mean 
square  error  of  the  data  (22:358;  48:864).  When  the 
interval  contains  the  number  zero,  meaning  that  the  samples 
may  have  the  same  mean,  the  comparison  is  not  considered 
significantly  different.  However,  if  zero  is  not  within  the 
interval,  then  the  comparison  is  significantly  different 
(35). 

Miller  states,  "no  method  is  uniformly  best  for  all 
sets  of  data",  because  each  multiple  comparison  technique  is 
based  on  a  different  set  of  assumptions  (49:236).  The 
Bonferroni  multiple  comparison  procedure  controls  the 
experiment-wise  error  rate  by  pre-selecting  the  samples  to 
be  compared.  Also,  this  technique  does  not  permit  data 
snooping,  which  is  basing  the  comparisons  on  data  which  is 
not  pre-selected  (35). 
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Hq:  ui=  (J2:=l-l3  no  significant  difference  exits) 

Ha:  at  least  one  does  not  equal  0 

(a  significant  difference  does  exit) 

Test  Statistic: 

F=SST/(p-l) 

SSE/(n-p)  where, 

SST=Nxl(Xxl-X)2  +Nx2  (Xx2-X)2  +...Nxn(Xxn-X)2 

SSE=Z[(Xx1-Xx1)2]+Z[(Xx2-Xx2)2]+. . ^[(Xxn-Xxn)2] 

p=number  of  populations 
n=number  of  observations  in  the  sample 
Rejection  Region: 

F>Fa 

Conclusion:  Reject  H0  if: 

1)  test  statistic  lies  in  rejection  region 

OR 

2)  p-value<a 

Figure  9.  Hypothesis  Test  for  the  One-way  ANOVA  (28) 

This  procedure  can  be  used  even  if  the  ANOVA  does  not 
identify  any  significant  differences  between  the  samples. 
This  is  illustrated  by  the  fact  that  at  a  .05  level  of 
significance,  there  is  still  a  5%  chance  that  the  null 
hypothesis  will  be  rejected  when  it  is  true.  Therefore, 
there  is  a  5%  chance  that  a  significant  difference  exists, 
but  was  not  identified  (35). 
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The  pooled  variance  t-test  ie  used  to  determine  if  a 
significant  difference  exists  between  two  samples.  This 
test  uses  the  t-test  statistic,  which  measures  the 
difference  in  the  sample  means  minus  the  hypothesized 
difference,  divided  by  the  quantity,  pooled  variance  times  a 
function  of  the  two  sample  sizes.  To  apply  this  test,  three 
assumptions  must  be  met  (35): 

1.  Samples  come  from  normal  populations. 

2.  Random  and  independent  sampling  was  used. 

3.  The  variance  between  the  samples  is  equal. 
Figure  10  describes  the  pooled  variance  t-test  hypothesis 
test . 

Non-par ame trie  Statistics.  A  non-par ame trie 
statistical  test  is  a  test  "whose  model  does  not  specify 
conditions  about  the  parameters  of  the  population  from  which 
the  sample  was  drawn"  (60:31).  Non-parametric  statistical 
techniques  do  not  rely  on  as  many  assumptions  as  do 
parametric  techniques.  As  a  result,  your  conclusions  do  not 
require  as  many  qualifications  either.  A  suitable 
conclusion  drawn  after  using  a  non-parametric  technique  may 
say,  "Regardless  of  the  shape  of  the  population(s) ,  we  may 
conclude  that..."  (60:3).  Non-parametric  techniques  are 
often  referred  to  as  "ranking  tests"  or  "order  tests" 
because  they  focus  on  the  rank  or  order  of  the  scores,  not 
on  the  numerical  values  (48: 946-947; 60: 3) . 
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Hc:  mi~U2=0  (no  significant  difference  exists) 

Ha;  u1-U2=/0  (a  significant  difference  exists) 
Test  Statistic: 

1/2 

t= [ (Xl-x2 )-D0]/[p( l/ni+l/n2 ) ] 

where:  x=avg.  score  for  sample  1  or  2 

D0=hypo the sized  difference  (0  for  small  samples) 

n=sample  size  of  sample  1  or  2 
P=[(n1-l)s5+(n2-l)8^]/[n1-n2-2] 
s=standard  deviation  of  sample  1  or  2 


Rejection  Region: 


t>ta/s 


t<ta/2 


Conclusion:  Reject  H  if: 


1)  test  statistic  lies  in  rejection  region 

OR 


2)  p-value<a 


Figure  10.  Hypothesis  Test  for  the  Pooled  Variance 
t-Test  (35) 


Siegel  states  four  primary  advantages  to  using  non- 
parametric  techniques.  They  are: 

1.  The  shape  of  the  distribution  is  not  important. 

2.  The  scores  do  not  have  to  be  numerically  exact,  as 
long  as  they  are  ranked. 

3.  Non-parametric  techniques  are  generally 
computationally  simpler  than  their  parametric  counterparts 

4.  Non-parametric  techniques  are  very  useful  when  you 
have  a  small  sample  size  (60:vii). 
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The  K-W  ANOVA  is  the  non-parametric  technique  which  is 
analogous  to  the  parametric  One-Way  ANOVA.  This  test  is 
used  when  you  have  independent  random  samples  (60:161). 

This  test  will  tell  you  if  the  independent  samples  are  from 
different  populations. 

If  the  experimental  design  has  three  or  more 
treatments,  this  test  can  only  determine  if  a  significant 
difference  exists,  but  can  not  locate  the  significant 
difference  (60:184-186).  Figure  11  shows  the  hypothesis 
test  for  the  K-W  ANOVA. 


HQ:  k  samples  come  from  the  same  population 

(no  significant  difference  exists) 

Ha:  at  least  one  of  the  k  samples  comes  from 

a  different  population  (a  significant 
difference  exists) 

Test  Statistic: 

H=  12  SR-,  2-3(N+l) 

N(N+l)  J  nj  where, 

k= number  of  samples 

nj=number  of  cases  in  the  jth  sample 
N=Znj ,  the  number  of  cases  in  all  samples 
combined 

Rj=sum  of  ranks  in  the  jth  sample 
Rejection  Region:  H>Ha 
Conclusion:  Reject  HQ  if: 

1)  test  statistic  lies  in  rejection  region 

OR 

2)  p-value<a 


Figure  11.  Hypothesis  Test  for  the  Kruskal-Wallis 
One-way  ANOVA  by  Rank  (60:185-186) 
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When  using  parametric  techniques,  the  One-way  ANOVA  is 

used  to  determine  if  a  significant  difference  exists  between 

any  of  the  treatments,  and  a  multiple  comparison  technique 

is  used  to  identify  which  specific  treatments  are 

significantly  different  from  one  another.  Since  there  is 

not  a  non-parametric  equivalent  to  the  multiple  comparison 

technique,  a  non-parametric  two-sample  test  should  be  used 

to  determine  which  treatments  are  significantly  different. 

However,  Siegel  makes  a  very  important  point: 

Only  when  an  over-all  test  (such  as  the  K-W  ANOVA) 
allows  us  to  reject  the  null  hypothesis. . .are  we 
justified  in  employing  a  procedure  for  testing  for 
significant  differences  between  any  two  of  the  k 
samples.  (60:160) 

If  a  non-parametric  two-sample  test  is  used  without 

first  conducting  an  over-all  test,  the  probability  of  a  Type 

I  error  is  increased,  as  is  the  probability  of  rejecting  the 

null  hypothesis  when  it  is  actually  true.  Siegel  says: 

It  can  be  shown  that,  for  five  samples,  the  probability 
that  a  two-sample  statistical  test  will  find  one  or 
more  'significant'  differences,  when  alpha=.05,  is 
p=.40.  That  is,  the  actual  significance  level  in  such  a 
procedure  becomes  alpha=.40.  (60:159-160) 

The  Mann-Whitney  U  test  is  a  two-sample  test  which  is 

used  to  determine  whether  two  independent  groups  have  been 

drawn  from  the  same  population.  When  the  data  is  at  least 

ordinal,  this  test  is  one  of  the  most  powerful  non- 

parametric  tests,  and  is  a  good  alternative  to  the 

parametric  t-test  (60:116).  Figure  12  illustrates  the 

hypothesis  test  for  the  Mann-Whitney  U  test. 
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HQ:  samples  come  from  the  same  population 
(no  significant  difference  exists) 

Ha:  samples  come  from  different  populations 
(a  significant  difference  exists) 

Test  Statistic: 

U=n^n2  +n-|(n-j+l)-R 
2 

ui  =nln2-^  where , 

n^=number  of  cases  in  the  smaller  sample 
n2=number  of  cases  in  the  larger  sample 

R=sum  of  the  ranks  assigned  to  the  group  whose 
sample  size  is  n^ 


Rejection  Region: 


1 


Reject  H0  if  the  smallest  value,  either  U  or  U 

<the  critical  value  of  U  from  the  Mann-Whitney 
table 


Figure  12.  Hypothesis  Test  for  the  Mann-Whitney 
U  Test  (60:118-120) 


Rfi_e;arch  Hypothesis 

The  Xerox  corporations'  United  States  marketing  group 
CAI  manager  said,  "if  it  is  used  appropriately,  CBT  can  be 
mor^  effective  than  many  more  traditional  training  methods" 
(40.22).  This  belief  parallels  the  fundamental  research 
hypothesis  for  this  study,  which  states  that  CBT  is  a  more 
useful  learning  tool  than  the  three  skill  level  CDC  for 
direct  duty  assignment  (DDA)  Material  Storage  and 
Distribution  Specialists,  AFSC  645X1.  In  other  words,  the 


65 


direct  duty  645Xls  who  use  CBT,  will  more  effectively  gain 
the  knowledge,  understanding,  and  skills  necessary  to 
achieve  their  three  skill  level,  as  compared  to  the  direct 
duty  645Xls  who  receive  the  traditional  645X1  CDC  training. 

Independent  Variable 

The  type  of  training  which  is  administered  to  the  test 
subjects  comprises  the  independent  variable.  The  test 
subjects  received  one  of  three  possible  treatments.  The 
first  treatment  is  the  CBT  module,  the  second  treatment  is  a 
text  training  module,  and  the  third  treatment  represents  the 
control  group.  The  subjects  in  the  control  group  did  not 
receive  any  training,  but  were  administered  the  post-test 
and  the  task  analysis. 

The  CBT  module  was  developed  using  the  QUEST  authoring 
system,  version  3.0.  The  text  training  module  is  composed 
of  excerpts  from  the  64531  CDC  Apprentice  Materiel 
Facilities  Specialist  (1989),  which  correspond  to  the 
objectives  used  in  the  CBT  module. 

Dependent  yard. able 

The  dependent  variable,  which  is  a  function  of  the  type 
of  training  administered  to  the  test  subjects,  represents 
the  degree  of  learning  which  the  test  subjects  will 
demonstrate  as  a  result  of  receiving  training.  Learning 
will  be  measured  in  two  ways: 
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1)  using  the  test  subjects'  post- test  scores,  and 

2)  observing  their  performance  in  a  task  analysis  (this  is 
explained  in  more  detail  under  the  subheading.  Post  Test ) . 

Experimental  Design 

The  post-test-only  control  group  design  will  be  used  to 
conduct  this  experiment.  Typically,  the  post-test-only 
control  group  design  relies  on  the  use  of  simple  random 
sampling.  However,  the  stratified  random  sampling  technique 
was  used  for  this  research  project.  By  minimizing  between- 
group  variability,  this  method  reduces  the  probability  that 
other  factors  will  effect  the  experimental  results.  In 
other  words,  stratifying  the  test  subjects  helps  to  minimize 
the  effect  other  factors  may  have  on  the  experimental 
results.  Figure  13  describes  the  post- test-only  control 
group' design  as  it  relates  to  this  study. 


Selection 

criteria 

R 

R 

R 


Experimental 

treatment 

Xx 

Xj. 

nt 


Post- 
test , 
P 
P 
P 


where , 

R=random  assignment  of  test  subjects 

xx  =CBT  training  module 

x*=CDC  training  module 

nt=no  treatment  (control  group) 

P=post-test 


Figure  13.  Post-test  Only  Control  Group  Design 
(23:122). 
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In  terms  of  this  research,  the  test  subjects  will  be 
assigned  to  one  of  the  three  experimental  treatments  based 
on  their  AQE  test  scores  in  the  general  category.  The  Air 
Force  uses  the  AQE  test  score  to  determine  the  career  field 
eligibility  of  individuals  who  are  entering  the  Air  Force. 
Using  the  general  score  is  preferred  because  the  air  Force 
uses  it  as  a  common  criteria  for  entrance  into  the  645X0  and 
645X1  career  fields  (21: Attachment  37). 

New  students  in  the  Supply  Inventory  Management 
Specialist  Course,  AFSC  645X0,  at  the  3440th  TTC,  were  used 
to  make  inferences  about  the  population  of  Supply  Material 
Storage  and  Distribution  Specialists,  AFSC  645X1.  This 
situation  is  unavoidable  because  645Xls  do  not  receive 
formal  technical  training,  instead  they  are  assigned 
directly  to  their  duty  station  upon  completion  of  Basic 
Military  Training  School  (BMTS)  at  Lackland  AFB,  Texas. 

Due  to  the  BMTS  graduation  schedule,  and  the  relatively 
small  number  of  future  645Xls  who  graduate  from  BMTS  at  any 
given  time,  it  is  not  feasible  to  UBe  KITS  graduates  as  test 
subjects. 

This  research  includes  a  statistical  analysis  which 
shows  that  the  two  populations,  645Xls  and  645X0 
experimental  subjects,  are  not  significantly  different. 
Figure  14  summarizes  the  minimum  AQE  test  score  requirements 
for  selection  as  a  Supply  Inventory  Management  Specialist, 
AFSC  645X0,  and  a  Material  Storage  and  Distribution 
Specialist,  AFSC  645X1. 
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Research  Procedures 


This  section  describes  the  procedures  used  to  conduct 
this  experiment.  The  logistical  limitations  imposed  by  this 
study,  as  well  as  the  details  of  how  the  stratified  random 
sampling  technique  was  implemented,  are  discussed  in  detail. 


AFSC 

Minimum  score 

Catecorv 

645X0 

45 

Admin,  or 

43 

General 

645X1 

30 

General 

Figure  14.  Minimum  AQE  Test  Scores  for  Entry  in 
AFSC  645X0  and  645X1  (21) 


In  addition,  the  data  collection  procedures  are  explained, 
followed  by  a  discussion  of  the  post-test.  This  section 
ends  with  a  discussion  of  the  personal  information 
questionnaire,  and  the  methods  used  for  data  analysis. 

Logistical  Limitations .  Although  very  straightforward 
in  design,  this  research  project  poses  an  interesting 
logistical  challenge,  which  has  three  root  causes.  The 
first  cause  is  the  geographic  distance  between  the  authors, 
assigned  to  Wright-Patterson  AFB,  OH,  and  the  test  subjects, 
assigned  to  the  3440th  TTC.  The  second  cause  is  a  result  of 
the  test  subjects'  limited  availability.  They  were  only 
available  for  a  maximum  of  two  duty  days  before  beginning 
their  initial  technical  training.  To  minimize  the 
possibility  that  advance  knowledge  of  supply  theory  and 
procedures  could  introduce  any  bias,  the  experiment  must  be 
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conducted  before  the  test  subjects  receive  any  technical 
training. 

In  addition  to  their  limited  availability,  recent 
budget  cuts  have  reduced  the  number  of  students  available 
per  week,  from  approximately  40  students  to  approximately 
10-15  students.  Therefore,  the  experiment  was  conducted  on 
four  separate  occasions  to  ensure  that  an  adequate  sample 
size  was  taken. 

Details  ol  Sampling  Method.  Each  time  the  experiment 

was  conducted,  the  general  scores  from  the  AQE  test  for  the 

test  subjects  were  ranked.  This  ranked  data  was  divided,  or 

stratified,  into  four  quartiles.  The  top  25 %  of  the  scores 

were  assigned  to  the  first  quartile,  the  upper-middle  25%  of 

the  scores  were  assigned  to  the  second  quartile,  the  lower- 

middle  25%  were  assigned  to  the  third  quartile,  and  the 

lowest  25%  of  the  scores  were  assigned  to  the  fourth 

quartile.  Once  all  test  subjects  were  assigned  to  a 

specific  quartile,  the  subjects  within  each  quartile  were 

randomly  selected  and  assigned  to  one  of  the  three 

treatments;  CBT,  CDC,  or  the  control  group.  Figure  15 

describes  how  the  Btrata  are  segregated,  and  Figure  16 

identifies  the  test  subject  composition  in  each  treatment. 

The  optimal  sample  size  for  this  research  effort  is 

determined  by  applying  the  following  formula  (1:12): 

n=[N(z* )*p(l-p)]/[(N-l)*(d= )+(z* )*p(l-p)]  where; 
n=  sample  size  N=  population  Bize 

p=  maximum  sample  size  factor  d=  confidence  level 

z=  z-test  statistic 
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Cats  Collection.  This  section  explains  how  the  data 
was  collected.  Given  the  limited  amount  of  travel  time 
available  for  the  authors,  and  the  limited  availability  and 


Ranked 

Quart ile 

AGE  scores  (%) 

assigned 

76  to  100 

first 

51  to  75 

second 

26  to  50 

third 

0  to  25 

fourth 

Note:  Only  AQE  scores 

from  the  general 

category  were  used  for  sample 

stratification 

Figure  15.  Strata  Determination  for  Sampling  Technique 


Treatment 

Composition 

CBT 

1/4  from  first  quartile 
second  quartile 

* 

third  quartile 
fourth  quartile 

CDC 

1/4  from  first  quartile 
second  quartile 
third  quartile 
fourth  quartile 

Control 

1/4  from  first  quartile 
second  quartile 
third  quartile 
fourth  quartile 

Note:  Test  subjects  within  each  strata  will  be 

randomly  assigned 

to  the  treatments 

Figure  16.  Composition  of  Experimental  Treatments 


number  of  tests  subjects,  a  sound  logistics  plan  was 
critical  to  the  success  of  this  research.  All  details  were 
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coordinated  ahead  of  time  to  ensure  that  the  experiment 
flowed  smoothly. 

A  POC  was  established  at  the  3440th  TTC.  and  his  active 
participation  played  a  key  role  in  the  efficiency  with  which 
the  experiment  was  conducted.  The  POC  was  responsible  for 
the  preparation  of  many  details  such  as  scheduling  the  test 
subjects  in  advance,  notifying  them  of  their  appointments 
for  the  experiment,  and  ensuring  availability  of  adequate 
classroom  space  and  microcomputer  support.  Travel  to  Lowry 
AFB  was  planned  to  occur  during  the  weeks  when  a  maximum 
number  of  students  were  scheduled  to  begin  class. 

Upon  arrival  at  Lowry  AFB,  the  author  immediately 
determined  the  composition  of  each  treatment.  A  copy  of  the 
procedures  used  to  determine  treatment  composition  is  found 
in  Appendix  B.  After  determining  treatment  composition,  the 
CBT  program  was  loaded  into  the  microcomputers  and  adequate 
copies  of  the  CDC  text  material  and  the  post-test  were  made 
available.  Once  details  such  as  these  were  completed,  the 
experiment  was  set  to  begin. 

The  test  subjects  within  each  treatment  were  segregated 
from  those  in  the  other  treatments  for  the  duration  of  the 
experiment.  This  ensured  that  no  inter  or  intra-treatment 
interactions  occurred,  potentially  biasing  the  data.  The 
experiment  administrator  read  a  prepared  introduction  to 
each  treatment.  This  introduction  provided  the  test 
subjects  the  information  they  needed  to  complete  their 
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specific  treatment.  A  copy  of  the  prepared  introductions  is 
provided  in  Appendix  C. 

Once  the  prepared  introductions  were  read  to  the  test 
subjects,  they  began  their  respective  training  modules.  The 
test  subjects  in  the  control  group  immediately  received  the 
post-test.  When  all  of  the  test  subjects  in  the  CBT  and  CDC 
treatments  completed  their  training  modules,  they  were 
allowed  a  15  minute  break,  which  was  monitored  to  ensure 
that  no  information  was  transferred  between  individuals. 
After  the  break,  the  CBT  and  CDC  test  subjects  received  the 
post-test . 

Once  the  post-test  was  completed,  the  test  subjects 
were  taken  to  the  base  supply  squadron  where  they  were  asked 
to  perform  a  task  analysis.  Specifically,  each  test  subject 
was  given  a  piece  of  property  and  a  management  notice,  and 
asked  to  properly  stock  the  property.  The  students  were 
evaluated  for  their  knowledge  of  the  proper  procedures,  and 
the  sequence  with  which  the  procedures  should  be  performed. 
Once  a] 1  of  the  test  subjects  completed  the  task  analysis, 
the  data  collection  phase  of  the  experiment  was  complete. 

Post-test .  The  test  subjects  were  evaluated  in  two 
ways.  The  descriptive  post-test  provided  a  quantitative 
measurement,  which  was  used  to  measure  the  degree  of 
learning  which  took  place  during  the  CBT  and  CDC  treatments. 
Results  from  the  control  group  provided  a  baseline  from 
which  to  compare  the  scores.  The  mastery  test  contained  in 


73 


the  prototype  CBT  program  was  used  as  the  experimental  post¬ 
test.  Since  the  text  module  and  the  CBT  module  cover  the 
same  material,  the  prototype  CBT  program  mastery  test  was 
administered  to  all  of  the  test  subjects.  Using  the  same 
post-test  for  all  of  the  test  subjects  ensured  that  the 
degree  of  learning  was  measured  consistently  within  each 
treatment . 

The  task  analysis  provided  another  quantitative 
measurement.  It  identified  the  degree  to  which  the  test 
subjects  in  each  treatment  could  transfer  their  newly 
acquired  knowledge  to  a  "real-world"  scenario.  During  the 
task  analysis,  the  subjects  were  taken  into  the  supply 
warehouse  at  the  Lowry  AFB  Supply  Squadron,  and  read  a  short 
introduction  which  was  designed  to  help  them  understand  the 
scenario.  After  receiving  the  introduction,  they  were  given 
a  management  notice  and  a  piece  of  property,  and  asked  to 
properly  store  the  property  in  the  warehouse.  Further,  they 
were  asked  to  verbalize  the  steps  they  performed  so  that  the 
monitor  could  more  easily  understand  and  follow  their  logic. 
Figure  17  provides  a  copy  of  an  actual  management  notice 
which  was  used  during  the  task  analysis.  A  copy  of  the  task 
analysis  introduction  is  found  in  Appendix  D,  and  a  copy  of 
the  experimental  checklist  used  to  evaluate  the  test 
subjects  performance  is  found  in  Appendix  E. 

In  addition  to  recording  the  scores  from  the  post-test 
and  from  the  task  analysis,  the  length  of  time  the  test 
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subjects  used  to  complete  the  training,  testing,  and  task 
analysis  portions  of  this  experiment  were  measured.  This 
data  was  used  to  determine  the  effect  CBT  has  on  the  amount 
of  time  required  to  complete  these  functions. 
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Figure  17.  Management  Notice  Used  During  Task 
Analysis 


Questionnaire .  The  questionnaire  contains  questions 
designed  to  gather  demographic  information  from  the  test 
subjects.  Areas  such  as  educational  level  achieved, 
previous  computer  experience,  age,  etc.  were  considered. 
This  data  will  be  used  to  help  identify  correlations  with 
the  dependent  variable,  should  the  results  prove 
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inconclusive,  and  to  provide  demographic  information  from 
the  test  subjects.  A  copy  of  the  questionnaire  is  found  in 
Appendix  F. 

Data  Analysis .  This  section  describes  the  parametric 
and  non-parametric  statistical  tests  used  to  analyze  this 
data.  The  software  program  Stat istix  II :  An  Interactive 
Statistical  Analysis  Program  for  Microcomputers  (Statistix), 
was  used  to  conduct  the  statistical  tests. 

If  parametric  statistical  techniques  were  used,  the 
software  program  titled.  Statistical  Analysis  Program  (SAS) 
could  be  used  to  perform  the  data  analysis.  SAS's  PROC 
ANOVA  command  would  be  used  to  conduct  a  one-way  ANOVA. 

Also,  the  /BON  option  associated  with  the  PROC  ANOVA  command 
would  be  used  to  perform  Bonferroni's  procedure  for  multiple 
comparisons  (35;  59: 224; 236).  Statistix  would  be  used  to 
conduct  Bartlett's  test  for  equality  of  variances  between 
the  samples  used  to  conduct  the  pooled  variance  t-test. 
Bartlett's  test  would  also  be  used  to  test  equality  of 
variances  between  the  treatment  data  in  the  one-way  ANOVA 
(52:5.56). 

The  pooled  variance  t-test  would  be  used  to  compare  the 
AQE  test  scores  in  the  administrative  category,  between  a 
sample  of  645X0  experimental  subjects,  and  a  random  sample 
of  newly  assigned  645Xls.  The  Bonferroni  procedure  would  be 
used  to  identify  which  treatments  were  significantly 
different,  regardless  of  the  results  of  the  one-way  ANOVA. 
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If  non-parametric  techniques  are  used,  Statistix  will 
be  used  to  compute  the  K-W  ANOVA,  the  Mann-Whitney  U  Test, 
the  pooled  variance  t-test,  and  any  other  tests  deemed 
appropriate . 

Review  ol  Experimental  Methodology 

From  the  very  beginning,  the  purpose  of  this  research 
was  to  objectively  examine  the  hypothesis  that  CBT  is  a  more 
effective  training  tool  than  CDCs  for  training  DDA  supply 
warehousemen.  The  authors  directed  a  great  deal  of  effort 
towards  developing  a  methodology  which  would  allow  a  clear 
examination  of  the  experimental  hypothesis.  They  went  to 
extreme  lengths  to  ensure  positive  control  of  internal  and 
external  validity,  and  to  eliminate  the  negative  effects  of 
potential  biasing  elements.  As  a  result,  the  authors  are 
very  confident  that  the  experimental  methodology  which  they 
applied  is  sound,  and  that  the  results  of  this  research  are 
convincing. 

There  is  not  a  single  geographic  location  where  an 
acceptable  number  of  new  Air  Force  DDA  supply  warehousemen 
are  located  at  any  one  time.  Therefore,  the  authors  chose 
to  use  airmen  who  were  preparing  to  enter  the  Supply 
Specialist  technical  training  course,  AFSC  645X0,  at  Lowry 
AFB,  as  the  test  subjects.  The  choice  to  use  645X0s  for  a 
study  aimed  at  645Xls  immediately  raises  some  understandable 
doubts  about  the  generalizability  of  the  experimental 
results.  However,  the  authors  have  closely  examined  the 
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selection  criteria  which  the  Air  Force  uses  to  select  airmen 
for  entrance  into  the  645X1  and  645X0  career  fields,  and 
they  provide  a  statistical  analysis  which  supports  the  idea 
that  the  two  populations  are  not  significantly  different 
from  one  another. 

The  authors  applied  the  post-test-only  control  group 
design  to  this  research,  using  three  experimental 
treatments.  The  test  subjects  in  the  first  treatment  were 
trained  using  a  CBT  module  which  was  designed  by  the  Lowry 
AFB  CBT  team  and  developed  by  AFLMC.  The  test  subjects  in 
the  second  treatment  were  trained  using  material  extracted 
from  the  64531  CDC.  The  material  which  was  extracted  from 
this  CDC  was  very  carefully  chosen  to  match  the  same 
training  objectives  that  the  CBT  module  uses.  The  test 
subjects  in  the  third  treatment,  the  control  group,  did  not 
receive  any  training. 

After  the  training  sessions  were  conducted,  all  of  the 
test  subjects  were  administered  an  identical  post-test  which 
measured  their  knowledge  level  of  the  training  objectives. 
Immediately  following  the  post-test,  the  test  subjects  were 
taken  to  the  base  supply  squadron  where  they  were  given  a 
realistic  scenario,  and  asked  to  perform  the  task  for  which 
they  were  trained.  Specifically,  they  were  given  a 
management  notice  and  a  pieje  of  property  and  asked  to 
perform  the  necessary  steps  to  properly  store  the  property 


in  the  warehouse. 


As  a  result  of  this  methodology,  it  is  possible  to 
objectively  measure  the  amount  of  cognitive  knowledge  the 
test  subjects  gained  from  their  training,  and  to  identify 
how  transferable  that  training  was  to  an  actual  work 
situation.  Also,  the  training  time  for  each  treatment  was 
easily  measured,  and  these  times  were  compared  to  the 
results  of  the  post-test  and  the  task  analysis. 

Summary 

This  chapter  discussed  how  the  research  hypothesis  was 
objectively  evaluated.  The  introduction,  which  explained 
why  an  experimental  design  was  chosen,  was  followed  by  a 
review  of  the  methodological  literature.  This  literature 
review  summarized  the  current  literature  as  it  pertains  to 
research  methods,  design,  end  the  statistical  techniques 
which'  may  be  used  for  this  study. 

The  research  hypothesis  was  restated,  and  the 
independent  and  dependent  variables  were  identified  and 
discussed.  Also,  specific  details  of  the  experimental 
design  and  the  research  procedures,  were  followed  by  a 
discussion  of  the  sampling  technique  used.  Finally,  the 
specific  statistical  tests  which  were  to  be  used  were 
discussed. 
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IV.  Data  Analysis 


Introduction 

This  chapter  will  address  each  of  the  seven 
investigative  questions  originally  posed  in  Chapter  I. 
Included  in  the  investigative  questions  is  an  analysis  of 
demographic  information,  which  is  offered  to  gain  some 
insight  into  the  test  subjects'  characteristics.  Finally, 
the  data  is  summarized  to  provide  a  brief,  logical,  and 
cohesive  statement  of  the  results  of  this  research. 

This  chapter  will  provide  a  summary  of  all  hypothesis 
tests  performed.  The  actual  computer  output  for  each 
hypothesis  test  was  produced  using  the  Statistix  software 
package,  and  is  contained  in  Appendix  G. 

Invest iRat ive  Question  Ml 

What  are  the  current  base  level  training  methods  for 
DDA  supply  warehousemen,  AFSC  645X1? 

Telephone  interviews  were  used  to  determine  the  present 
training  methods  currently  used  throughout  the  Air  Force  for 
training  DDA  645Xls. 

CMSgt  Lewis,  Directorate  of  Supply,  Headquarters 
Strategic  Air  Command  (SAC),  said  that  most  SAC  units  are 
relying  on  CDCs  to  train  their  DDA  supply  warehousemen.  To 
supplement  the  CDCs,  some  SAC  units  have  obtained  portions 
of  the  training  plan  from  the  discontinued  AFSC  645X1 
Initial  Training  Course,  which  was  formerly  taught  at  Lowry 
AFB.  CMSgt  Lewis  stated  that  SAC's  failure  rate  for  the 
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64531  CDC  end-of-course  examination  has  risen  since  the 

initial  645X1  course  was  terminated  at  the  3440th  TTC  (44). 

Jack  Hoskins,  Training  Section  supervisor  at 

Plattsburgh  AFB,  New  York,  stated: 

The  DDA  personnel  at  Plattsburgh  Air  Force  Base  use  the 
64531  CDC's  for  their  initial  training  in  storage  and 
distribution  procedures.  This  is  the  standard  training 
for  645X1  personnel  throughout  SAC.  (36) 

Linda  Neri,  Chief  of  Resources,  Headquarters  Tactical 

Air  Command  (TAC),  commented  that  it  is  TAC's  policy  to 

allow  the  base  level  training  sections  to  develop  and 

implement  training  programs  for  DDA  645X1  personnel.  She 

believes  that  most  initial  training  for  DDA  645X1  personnel 

at  base  level  consists  of  a  combination  of  the  64531  CDCs 

and  OJT  ( 51 ) . 

Mrs.  Mary  Snook,  Training  Section  supervisor  at  Myrtle 
Beach  AFB,  SC,  stated  that  DDA  645Xls  use  the  64531  CDCs  for 
their  initial  training.  PCs  are  used  to  present  text  from 
the  CDCs  to  the  trainees.  The  PC  is  alBO  used  as  a  tool  to 
pre-test  the  trainees  before  taking  the  end-of-course  test 
(64) . 

Mrs.  Snook  indicated  that  the  Materiel,  Storage  and 
Distribution  Branch  did  develop  their  own  training  program 
for  the  DDA  645Xls.  Upon  arrival,  the  trainees  are  rotated 
through  each  of  the  five  sections  in  the  branch,  spending 
one  week  in  each  section,  before  being  assigned  to  a  duty 
section.  Each  section  supervisor  has  a  written  training 
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plan  which  identifies  the  key  areas  in  which  the  trainee 
should  receive  training  (64). 

CMSgt  David  Reeves,  Supply  Training  Division, 
Headquarters  Material  Airlift  Command  (MAC),  stated  that  his 
division  developed  and  updated  slide/audio  training 
presentations  in  all  areas  of  initial  supply  training  since 
1985,  including  warehouse  location  and  storage  procedures. 
These  programs  ran  for  a  duration  of  30-45  minutes.  They 
were  distributed  to  all  base  level  training  sections 
throughout  MAC.  These  programs  were  used  in  conjunction 
with  CDCs  and  OJT  to  train  newly  assigned  personnel  (59). 

In  September  1989,  funding  and  manpower  cuts  forced  the 
division  to  stop  development  and  updates  of  the  training 
slide  programs.  Base  level  training  sections  were  directed 
to  use  the  programs  until  they  became  obsolete  due  to  new 
procedure  implementation  and/or  manual  changes.  When  the 
programs  are  no  longer  useful,  new  DDA  645Xls  will  be 
trained  using  the  64531  CDCs  and  OJT  (59). 

SSgt  Chester  Miloisock,  NCOIC  Base  Supply  Training 
Section,  Scott  AFB,  Illinois,  commented  that  primary 
training  for  DDA  645Xls  is  accomplished  by  using  the  64531 
CDCs  and  OJT.  He  said  a  PC  is  used  for  administering  the 
Volume  Review  Exercises  in  preparation  for  end-of-course 
tests  (50). 
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Investigative  Question  £2 


Do  the  645X0  experimental  subjects  represent  a 
statistically  similar  population  compared  to  newly  assigned 
DDA  645Xls  (i.e.  are  the  means  of  their  AQE  scores  in  the 
general  category  similar)? 

Before  applying  any  statistical  tests,  the  AQE  scores 
of  the  entire  sample  of  test  subjects  (AFSC  645X0)  must  be 
compared  to  AQE  scores  taken  from  the  645X1  population. 

This  is  done  to  ensure  that  the  two  populations  are  not 
significantly  different  from  one  another. 

AQE  scores  for  the  period  of  1  Jan  89-31  Dec  89  were 
obtained  from  the  HRLDB.  A  total  of  439  DDA  645X1  personnel 
took  the  AQE  test  during  the  time  period  of  1  Jan  89- 
31  Dec  89.  In  order  to  determine  the  sample  size  required 
for  a  95%  confidence  level,  the  sample  size  formula  cited  in 
Chapter  III  was  used.  The  variables  in  the  formula  were  set 
as  follows;  N=439,  p=.50,  d=.05,  and  z=1.96.  The 
calculation  indicated  that  a  total  sample  size  of  206  was 
required.  A  random  sample  of  206  scores  was  selected  from 
the  439  645X1  AQE  scores,  and  compared  to  the  AQE  scores  of 
the  49  experimental  subjects.  In  order  to  determine  if 
parametric  tests  could  be  used  for  this  comparison  of  the 
AQE  scores,  the  assumptions  that  the  samples  are  normally 
distributed,  and  equality  of  variances  had  to  be  tested. 

To  satisfy  the  requirement  that  the  samples  are 
normally  distributed  the  Central  Limit  Theorem  was  imposed. 
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Since  there  were  49  subjects,  and  the  645X1  sample  size  was 
206,  both  samples  met  the  generally  accepted  sample  size  of 
30  in  order  to  impose  the  CLT.  Therefore,  the  samples  have 
satisfied  the  assumption  of  being  normally  distributed. 

Next,  the  samples  were  tested  for  equality  of  variance 
using  Bartlett's  Test  of  Equal  Variances.  The  null 
hypothesis  that  the  variances  of  the  two  samples  are  equal 
was  rejected  (P  value=0 . 0319 )  at  a  .05  significance  level. 

Since  the  samples  were  each  found  to  be  normally 
distributed,  but  had  unequal  variances,  the  assumptions 
required  for  the  use  of  parametric  tests  were  not  satisfied. 
Therefore,  to  determine  if  there  is  a  difference  in  AQE 
general  scores  for  the  experimental  subjects  and  DDA  645Xls, 
the  non-parametric ,  Mann-Whitney  U  Test  was  used. 

Using  the  Mann-Whitney  U  Test  for  the  comparison  of  AQE 
scores  of  the  two  samples,  a  P  value=0 . 7189  is  obtained. 
Therefore,  at  a  .05  significance  level,  the  conclusion  is 
not  to  reject  the  null  hypothesis;  there  is  no  significant 
difference  in  the  AQE  scores  of  the  two  samples.  The 
statistical  tests  used  to  analyze  this  investigative 
question  can  be  found  on  page  132,  Appendix  G. 

Investigative  Question  13 

Are  there  any  significant  differences  between  the  test 
subjects'  AQE  scores  from  the  three  experimental  treatments? 

The  AQE  scores  in  the  general  category  were  used  as  the 
criteria  to  stratify  the  treatments.  Stratification  ensures 
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that  the  test  subjects  are  represented  proportionately  in 
each  of  the  three  samples. 

Using  the  K-W  ANOVA  test  and  a  .05  significance  level, 
no  significant  differences  were  found  between  the  test 
scores  in  any  of  the  three  treatments  (P  Value= . 7181 ) .  The 
statistical  test  used  to  analyze  t,his  investigative  question 
can  be  found  on  page  133,  Appendix  G.  Table  1  contains  the 
AQE  scores  for  the  test  subjects  in  each  treatment. 

Investigative  Question  M 

Do  the  test  subjects  learn  more  when  trained  usins  CBT? 

Using  the  K-W  ANOVA  test  to  make  the  over-all 
comparisons  of  the  post-test  scores,  the  null  hypothesis  was 
rejected  (P  Value= . 0000 ) .  Therefore,  there  is  a  highly 
significant  difference  between  one  or  more  of  the  treat¬ 
ments’  post-test  scores:  CBT=75.6,  Text=54,  Control=32 . 2 . 

The  Mann-Whitney  U  test,  a  two-sample  test,  was  used  to 
identify  where  the  significant  difference(s)  were.  First, 
the  post-test  scores  between  the  text  treatment  and  the  CBT 
treatment  were  compared  using  a  .05  significance  level. 

This  test  identified  the  existence  of  a  highly  significant 
difference  (P  Value= . 0006 ) .  Second,  the  post-test  scores 
between  the  text  treatment  and  the  control  group  were 
compared  using  the  same  significance  level,  and  a  highly 
significant  difference  between  these  treatments  was 
identified  (P  Value= . 0001 ) .  Finally,  the  CBT  treatment  was 
compared  to  the  control  group  and,  again,  a  highly 
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significant  difference  between  these  treatments  was 


identified  (P  Value= . 0000  ) .  The  statistical  tests  used  to 
analyze  this  investigative  question  can  be  found  on  pages 
134-135,  Appendix  G.  Table  1  contains  the  post-test  scores 
for  the  test  subjects  in  each  of  the  three  treatments. 

TABLE  1 


POST-TEST  SCORES,  TASK  ANALYSIS  SCORES,  AQE  SCORES, 
AND  DESCRIPTIVE  STATISTICS 


i 

Text 

Treatment 

CBT 

Treatment 

Control 

Group 

Igjal 

Task 

AQE 

Iasi 

Task 

AQE 

Taai 

Task 

AQE 

38 

0 

55 

77 

80 

74 

27 

0 

48 

46 

40 

64 

85 

40 

80 

42 

0 

59 

62 

60 

84 

45 

60 

48 

25 

0 

53 

69 

70 

68 

77 

80 

42 

42 

0 

57 

50 

0 

37 

69 

50 

62 

42 

0 

62 

42 

10 

55 

69 

70 

33 

18 

0 

55 

36 

60 

57 

100 

30 

53 

25 

0 

62 

42 

10 

70 

92 

70 

94 

42 

0 

64 

62  ' 

60 

78 

92 
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0 

42 

50 
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42 
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90 
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54.0 
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9  32.2 
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Investigative  Quest ion  15 


Do  the  test  subjects  perform  the  task  analysis  better 
when  trained  using  CBT? 

Using  the  K-W  ANOVA  test  on  the  task  analysis  scores  at 
the  .05  level,  a  highly  significant  difference  was 
identified  between  the  task  analysis  scores  of  one  or  more 
of  the  treatments  (P  Value=.0000) .  Next,  the  Mann-Whitney  U 
test  was  used  to  identify  which  treatments  were 
significantly  different.  Comparing  the  text  treatment  to 
the  CBT  treatment  produced  a  highly  significant  difference 
(P  Valuer . 0012 ) .  Comparisons  of  the  text  treatment  to  the 
control  group  (P  Value= . 0004 )  and  the  CBT  treatment  to  the 
control  group  (P  Valuer. 0000)  also  identified  highly 
significant  differences  at  the  .05  level.  The  statistical 
tests  used  for  this  investigative  question  can  be  found  on 
pages  136-137,  Appendix  G.  Table  1  also  contains  the  task 
analysis  scores  for  the  test  subjects  in  each  of  the  three 
treatments . 

Investigative  Question  M 

When  the  test  subjects  were  trained  using  CBT,  was  the 
training  time  reduced? 

Since  the  control  group  did  not  receive  any  training, 
the  Mann-Whitney  U  test  was  used  to  see  if  a  significant 
difference  existed  between  the  CBT  treatment  and  the  Text 
treatment.  This  test  clearly  shows  that  a  highly 
significant  difference  does  exist  between  these  two 
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treatments  (P  Value=.0010)  at  the  .05  level.  The 
statistical  test  used  to  analyze  this  investigative  question 
can  be  found  on  page  138,  Appendix  G.  Table  2  contains  the 
training  times  for  the  test  subjects  in  the  CBT  and  Text 
treatments . 


TABLE  2 

TRAINING  MODULE  TIMES,  POST-TEST  TIMES, 

TASK  ANALYSIS  TIMES  (MINS),  AND  DESCRIPTIVE  STATISTICS 


Text  Treatment  CBT  Treatment  Control  Group 
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Investigative  Question  £2 

Does  computer  experience  and  education  affect  the 
student's  comprehension  of  material  presented  via  CBT? 

The  next  section,  a  demographic  analysis,  will  be  used 
to  answer  this  investigative  question.  The  statistical 
tests  used  to  analyze  this  investigative  question  can  be 
found  on  pages  139-140,  Appendix  G. 

Demographic  Analysis  ol  Experimental  Subjects 

Using  the  questionnaire,  personal  information  was 
obtained  from  all  test  subjects.  This  section  describes  the 
results  of  this  information.  This  information  helps  provide 
an  insight  into  the  characteristics  of  the  test  subjects 
used  in  this  experiment,  and  may  provide  an  insight  into  the 
characteristics  of  individuals  entering  the  supply  career 
field.-  The  compiled  results  of  the  questions  contained  on 
the  questionnaire  are  provided  in  the  following  discussion. 

Figure  18  shows  that  57%  of  the  test  subjects  were  male 
and  43%  were  female . 


What  is  your  sex? 

Male 

Female 

28 

21 

Figure  18.  Demographics:  Sex 


Figure  19  shows  that  more  than  55%  of  the  test  subjects 
were  either  18  or  19  years  of  age,  and  more  than  24%  of  the 
subjects  were  either  20  or  21  years  of  age.  Therefore,  the 
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vast  majority  of  the  test  subjects  (79%)  range  from  18-21 
years  of  age. 


What 

is  your  age? 

18  or 

less  18-19  20-21 

22-23 

24-25 

26  or  over 

0 

27  12 

5 

3 

2 

Figure  19.  Demographics:  Age 


Figure  20  describes  the  educational  level  of  the  test 
subjects.  Over  44%  of  the  test  subjects  had  at  least  some 
college . 

Figure  21  illustrates  that  a  large  majority  of  the  test 
subjects  (82%)  had  some  experience  using  computers.  Another 
interesting  point  brought  out  by  the  questionnaire  is  that 
of  the  subjects  with  computer  experience,  over  20%  had  more 
than  3  years  of  computer  experience.  The  next  section  in 
Data  Analysis  will  determine  if  subjects  with  computer 
experience  did  better  in  the  experimental  measurements. 


What  is  your  highest 

educational  level? 

0 

27 

16 

3 

Bachelors  Decree 

3 

Masters  Degree 

0 

Figure  20.  Demographics:  Educational  Level 
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Do  you  have  any  computer  experience? 

Yes  No 

40  9 

How  long  have  you  been  using  computers? 

Less  than  6  mos.  6  mos.->l  vr.  1  yr.->2  yrB. 

15  6  5 

2  vrs.->3  vrs.  More  than  3  vrs. 

4  10 

Figure  21.  Demographics:  Computer  Experience 

Figure  22  shows  that  of  those  subjects  indicating  they 
had  computer  experience  over  62%  have  some  programming 
experience . 

Effect  Education  and  Computer  Experience .  In  order 
to  determine  whether  the  degree  of  education  or  computer 
experience  had  an  effect  on  the  test  subjects'  performance 

Do  you  have  any  programming  experience? 

■Jfefi .  No 

25  15 

Figure  22.  Demographics:  Programming  Experience 

on  the  experimental  measurements  in  the  CBT  group,  the  Mann- 
Whitney  U  test  was  used.  To  determine  the  education  effect, 
test  subjects  were  divided  into  two  groups  according  to 
their  responses  on  the  questionnaire.  The  first  group 
consists  of  the  test  subjects  whose  highest  educational 
level  was  a  high  school  diploma,  the  next  group  consists  of 
those  who  had  at  least  some  college  experience.  The  Mann- 
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Whitney  U  test  showed  that  there  was  no  significant 
difference  (P  Value=0.4510)  between  the  test  scores  of  the 
high  school  graduates  and  those  having  at  least  some  college 
education  at  the  .05  significance  level.  The  Mann-Whitney  U 
test  also  showed  there  was  no  significant  difference  between 
the  task  analysis  scores  of  the  test  subjects  with  a  high 
school  education  and  those  with  college  experience 
(P  Value  =  0.3657)  at  the  .05  significance  level. 

Next,  the  test  subjects  were  separated  into  groups  of 
those  with  six  months  or  less  computer  experience,  and  those 
with  more  than  six  months  of  computer  experience.  The  Mann- 
Whitney  U  test  showed  that  there  was  no  significant 
difference  between  the  group's  test  scores  and  task  analysis 
scores  (P  Values=0. 9616  and  0.4705  respectively)  at  the  .05 
significance  level. 

Summary  of  Demographic  Analysis.  The  results  of  the 
demographic  analysis  indicate  that  on  the  average,  test 
subjects  were  relatively  young  and  well  educated,  over  40% 
of  whom  had  some  college  experience  and  all  had  at  least  a 
high  school  diploma.  Most  of  the  test  subjects  had  some 
experience  using  computers;  many  with  over  2  years  of 
experience.  It  does  not  appear  as  if  education  level  or 
prior  computer  experience  had  an  effect  on  the  amount  of 
learning  by  the  CBT  test  subjects. 
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Summary  of  Data  Analysis 


This  chapter  provides  the  answers  to  the  seven 
investigative  questions  which  were  originally  posed  in 
Chapter, I .  This  summary  attempts  to  encapsulate  the 
experimental  results  in  a  logical  and  cohesive  manner. 

Since  the  termination  pf  the  initial  645X1  training 
course  at  Lowry  AFB,  most  Base  Supply  Squadrons  have  not 
been  provided  any  strict  guidance  on  how  to  train  their  DDA 
supply  warehousemen.  Many  squadrons  are  relying  on  the 
64531  CDCs  to  fill  the  training  void,  and  some  bases  have 
made  some  efforts  to  expand  645X1  training.  For  example. 
Myrtle  Beach  AFB  has  developed  a  training  program  for  their 
new  DDA  645Xls,  where  the  trainees  spend  one  week  in  each  of 
the  five  sections  in  the  Materiel,  Storage  and  Distribution 
Branch  before  being  assigned  to  a  permanent  section.  HQ  MAC 
distributed  training  slide/audio  training  programs 
throughout  the  command,  but  these  programs  are  rapidly 
becoming  obsolete. 

The  data  collected  from  the  HRLDB  showed  that  there  is 
not  a  statistically  significant  difference  in  the  AQE  scores 
in  the  general  category  for  new  645Xls  and  the  645X0s,  who 
were  used  as  the  experimental  subjects.  Therefore,  based  on 
the  selection  criteria  of  using  AQE  scores  in  the  general 
category,  the  population  of  645Xls  and  645X0s  used  as 
experimental  subjects  are  not  significantly  different  than 
the  actual  population  of  645Xls. 
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Statistical  analysis  has  shown  that  all  of  the  subjects 
in  the  three  experimental  treatments  are  similar  (i.e.  not 
significantly  different).  Therefore,  the  three  treatments 
are  "balanced" ;  meaning  that  they  are  represented  equitably 
in  teiLi8  of  AQE  score  distribution. 

The  post-test  scores  for  the  CBT  and  Text  treatments 
were  significantly  different  from  one  another,  with  the  mean 
score  for  the  CBT  treatment  being  21.6  points  higher  than 
the  mean  score  of  the  text  treatment.  Also,  the  task 
analysis  scores  for  the  test  subjects  in  the  CBT  and  Text 
treatments  were  significantly  different  from  one  another, 
with  the  mean  score  for  the  CBT  treatment  being  32.4  points 
higher  than  the  Text  treatment.  Further,  the  training  times 
for  the  test  subjects  in  the  CBT  and  Text  treatments  were 
signif ioantly  different  from  one  another,  with  the  mean  time 
for  the  CBT  treatment  being  18.0  minutes  shorter  than  that 
of  the  Text  treatment. 

The  analysis  of  the  effect  of  computer  experience  and 
educational  level  of  the  subjects  in  the  CBT  group  revealed 
that  a  higher  level  of  education  and\or  computer  experience 
does  not  necessarily  enable  the  test  subjects  to  perform 
better  on  either  the  post-test  or  task  analysis. 
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3L.  Conclusions.  Recommendations ,  and  Summary 

Introduction 

Current  literature  clearly  supports  the  hypothesis  that 
CBT  is  an  effective  training  technique  or  tool.  However, 
the  development  of  CBT  in  the  Air  Force  supply  community  is 
only  in  its  infancy,  and  most  of  those  involved  in  the 
process  have  little  or  no  experience  developing  CBT. 
Therefore,  it  is  important  to  determine  the  effectiveness  of 
the  relatively  new  Air  Force  Supply  CBT  development  program 
before  making  any  long-term  commitments  towards  CBT 
development . 

In  order  to  clearly  determine  the  effectiveness  of  the 
CBT  development  program,  as  well  as  to  address  this  study's 
research  hypothesis,  this  chapter  will  state  the  final 
conclusion  for  each  investigative  question.  Also,  some 
recommendations  for  further  research  in  the  CBT  arena  are 
proposed,  some  lessons  learned  are  discussed,  and  a  summary 
of  the  research  methodology  and  the  experimental  results  are 
provided. 

ConcJLuBiQng 

This  research  led  to  some  very  interesting  and  very 
i  significant  results  which  strongly  support  the  use  of  CBT. 

Each  investigative  question  is  addressed  in  order. 
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Investigative  Question  pi.  What  are  the  current  base 


level  training  methods  for  DDA  supply  warehousemen,  AFSC 
645X1? 

For  the  most  part,  the  Air  Force  is  primarily  using  the 
64531  CDC  to  train  DDA  warehousemen.  Although  some  bases 
surveyed  have  developed  alternate  training  methods,  most 
rely  solely  on  the  CDC.  Also,  it  is  apparent  that  no 
efforts  have  been  made  higher  than  base  level,  to  develop  a 
standardized  training  program  for  DDA  645Xls. 

Investigative  Question  M2.  Do  the  645X0  experimental 
subjects  represent  a  statistically  similar  population 
compared  to  newly  assigned  DDA  645Xls  (i.e.  are  the  means  of 
their  AQE  scores  in  the  general  category  similar)? 

645X0s  do  not  represent  a  statistically  different 
population  than  645Xls,  in  terms  of  their  AQE  score  in  the 
general  category.  The  results  of  this  analysis  were  clearly 
significant  at  a  .05  level  of  significance.  Therefore,  it 
is  safe  to  assume  that  the  results  of  this  experiment  would 
not  be  significantly  different  had  the  test  subjects 
actually  been  DDA  645Xls. 

Investigative  Question  M2 .  Are  there  any  significant 
differences  between  the  test  subjects'  AQE  scores  in  the 
three  experimental  treatments? 

At  the  .05  level  of  significance,  the  K-W  ANOVA  clearly 
showed  that  there  are  no  significant  differences  between  any 
of  the  three  experimental  treatments.  Therefore,  it  is  safe 
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to  assume  that  the  three  treatments  are  representative  of 
the  same  population. 

Investigative  Question  MA-  Do  the  test  subjects  learn 
more  when  trained  using  CBT  versus  CDCs? 

The  test  subjects  in  the  CBT  treatment  learned 
significantly  more  than  the  test  subjects  in  the  text 
treatment  as  measured  by  the  post-test  with  the  CBT,  Text, 
and  Control  groups  achieving  mean  scores  of  75.6,  54.0,  and 
32.2  respectively.  The  results  of  the  statistical  analysis 
were  significant  at  a  .01  level  of  significance.  Therefore, 
CBT  can  significantly  improve  the  amount  of  learning  which 
occurs,  when  compared  to  text  (CDC)  training. 

Investigative  Question  #5 .  Do  the  test  subjects 
perform  the  actual  task  better  when  trained  using  CBT? 

At  a  .01  level  of  significance,  the  test  subjects  in 
the  CBT  treatment  (mean  score=67.1)  performed  significantly 
better  on  the  task  analysis  than  did  the  teBt  subjects  in 
the  text  treatment  (mean  score=34.7).  Therefore,  it  is 
concluded  that  CBT  can  significantly  improve  the  ability  of 
trainees  to  perform  the  tasks  for  which  they  are  trained. 

Inyegtigatlyg  Question  J*j6.  When  the  test  subjects  were 
trained  using  CBT,  was  training  time  reduced? 

A  comparison  of  the  CBT  and  text  treatments  at  a  .01 
level  of  significance,  showed  that  CBT  (mean  time=63.4 
mins. )  can  significantly  reduce  the  amount  of  time  required 
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for  training  versus  the  text  treatment  (mean  time=B1.4 
mins. ) . 

Investigative  Question  £2.  Does  computer  experience 
and  education  level  affect  the  student's  comprehension  of 
material  presented  via  CBT? 

At  a  .05  level  of  significance,  the  test  subjects  in 
the  CBT  treatment  did  not  score  significantly  different  on 
the  post-test  or  the  on  the  task  analysis,  regardless  of 
their  level  of  education.  Also,  prior  computer  experience 
did  not  significantly  improve  the  post-test  or  task  scores 
of  the  test  subjects  in  the  CBT  treatment. 

Practical  Significance .  Practical  significance  refers 
to  the  ease  with  which  one  can  scan  the  data  and  identify 
obvious  differences.  For  a  manager,  the  practical 
significance  of  the  data  can  provide  an  idea  of  the 
magnitude  of  the  results,  in  terms  which  are  more  easily 
understood.  In  terms  of  their  practical  significance,  the 
data  in  three  particular  areas  tends  to  stand  out  from  the 
rest  of  the  data. 

1.  The  mean  training  time  for  the  CBT  treatment  was 
approximately  63  minutes,  compared  to  81  minutes  for  the 
text  treatment.  On  the  average,  the  test  subjects  in  the 
CBT  treatment  completed  their  training  22%  faster  than  did 
the  test  subjects  in  the  text  treatment. 

2.  The  mean  post-test  scores  for  the  CBT  and  text 
treatments  were  approximately  76  and  54  respectively. 
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Therefore,  the  test  subjects  in  the  CBT  treatment  scored  28% 
higher,  on  average,  than  those  in  the  text  treatment. 

3.  The  mean  task  analysis  scores  for  the  CBT  and  text 
treatments  were  approximately  67  and  34  respectively.  As  a 
result,  the  average  CBT  test  subject  was  able  to  accomplish 
almost  7  of  the  required  steps  in  the  task  analysis  compared 
to  the  less  than  3.5  correct  steps  performed  by  the  average 
test  subject  in  the  text  treatment. 

Recommendat ions  for  Future  Research 

The  number  of  potential  studies  which  could  be 
performed  in  the  area  of  CBT  is  limited  only  by  one's  own 
imagination.  Since  this  study  and  others  have  shown  that 
CBT  is  an  effective  training  technique,  there  should  be 
additional  research  efforts  to  determine  which  elements  of 
CBT  have  the  greatest  impact  on  the  test  subjects'  rate  of 
learning.  For  example,  what  effect  does  varying  factors 
such  as  screen  design,  layout,  use  of  graphics,  program 
interaction,  remediation,  etc.  have  on  the  amount  of 
learning  accomplished  by  the  test  subjects?  Research  such 
as  this  can  help  identify  which  combinations  of  features 
work  well  to -ether  to  enhance  learning. 

Another  potential  area  for  future  research  would  be  to 
compare  the  effectiveness  of  CBT  modules  to  actual  classroom 
lectures  or  other  training  techniques.  This  could  be  done 
using  a  methodology  similar  to  the  one  used  for  this  study. 
Research  such  as  this  could  be  used  to  support  the 
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elimination  of  technical  training  courses,  or  to  support  the 
integration  of  CBT  modules  into  existing  technical  training 
courses. 

After  the  CBT  modules,  which  were  developed  by  the 
office  at  Lowry  AFB,  are  used  in  the  field  for  at  least  six 
months,  a  survey  should  be  conducted  to  examine  the  impact 
that  CBT  has  had  on  improving  the  knowledge  and  skills  of 
DDA  supply  warehousemen.  A  study  such  as  this,  when  coupled 
with  this  research,  could  provide  some  valuable  new 
information  about  the  long-term  effects  of  CBT. 

Experimental  Logistics  Lessons  Learned 

In  order  for  the  experiment  to  be  successful,  detailed 
advanced  planning  and  support  was  required  from  several 
organizations.  The  most  critical  organizations  providing 
support  were  the  3440th  TTC,  and  the  AFLMC.  This  section 
describes  the  support  these  organizations  provided  and 
lessons  learned  in  the  planning  and  implementation  of  this 
study. 

3440th  TTC  Support .  The  3440th  provided  pre-trip 
coordination  with  agencies  located  at  Lowry  AFB.  Capt  Brown, 
the  3440th  POC  provided  exceptional  support  at  Lowry  AFB. 
Without  his  assistance,  this  experiment  would  have  been  much 
more  difficult,  if  not  impossible,  to  complete. 

Coordination  with  four  base  organizations  was  required  prior 
to  arrival  at  Lowry  AFB:  the  3440th  Computer  Support 
Section,  3440th  Student  Training  Squadron,  Lowry  AFB  Supply 
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Squadron,  and  CBPO  Outbound  Assignments  Section.  Capt  Brown 
coordinated  closely  with  these  organizations  at  least  one 
week  before  the  authors'  site  visit  to  Lowry  AFB. 

Even  with  Capt  Brown's  prior  coordination,  some  minor 
scheduling  problems  did  occur.  However,  most  of  the 
problems  were  fairly  easily  overcome,  and  the  resultant 
support  provided  by  the  3440th  TTC  and  the  other  Lowry  AFB 
organizations  was  superb  and  greatly  contributed  to  the 
success  of  this  study. 

AFLMC  Support .  The  primary  support  required  from  the 
AFLMC  was  to  provide  the  CBT  lesson  and  post-test  for  the 
experiment.  AFLMC  authored  the  CBT  which  s  used  in  this 
experiment.  The  CBT  lesson  needed  to  be  transferred  from 
the  MERLIN  authoring  language  to  the  QUEST  authoring  system 
prior  to  release.  The  transfer  and  programming  of  the 
lesson  was  accomplished  at  AFLMC  from  Dec  89-  Mar  90  by  Capt 
Geasey . 

When  the  original  program  was  received  on  15  Mar  90, 
several  problems  with  the  program  were  encountered.  The  two 
major  problems  consisted  of  program  loading  difficulties  and 
an  incorrect  coding  of  a  question  in  the  post-test.  The 
loading  problems  were  a  result  of  the  software 
configuration.  In  order  for  the  program  to  load  properly 
the  authors  were  required  to  edit  some  of  the  configuration 
files  in  the  program.  The  error  in  the  post-test  question 
was  a  result  of  Capt  Geasey,  who  had  no  supply  experience  or 
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training,  incorrectly  linking  an  incorrect  response  as  the 
correct  response  for  question  number  eight  of  the  post-test. 
AFLMC  was  contacted  about  the  problems  with  the  lesson  and 
took  corrective  actions. 

A  corrected  lesson/test  was  received  from  AFLMC  on  10 
May  90,  just  days  prior  to  the  first  planned  trip  to  Lowry 
AFB.  The  updated  version  corrected  many  of  the  problems 
encountered  in  the  initial  software.  However,  additional 
editing  of  the  program  files  was  still  required  in  order  to 
bring  about  proper  program  loading. 

With  the  exception  of  the  program  loading  problems 
encountered  with  the  CBT  program,  overall  support  provided 
by  the  AFLMC  was  good.  Guidance  from  Lt  Col  Peterson  and 
the  additional  temporary  duty  (TDY)  funding  provided  proved 
to  be  invaluable  to  this  research  effort. 

Summary  of  Lessons  Learned.  There  were  two  areas 
concerned  with  carrying  out  this  research  that  could  have 
been  improved. 

First,  the  coordination  with  Lowry  AFB  organizations 
could  have  been  improved.  If  the  authors  could  have 
traveled  to  Lowry  AFB  prior  to  the  experimental  trips,  to 
personally  explain  the  intricacies  of  the  experiment  with 
the  organizations  involved,  most  scheduling  problems  and 
misunderstandings  could  have  been  avoided.  Unfortunately, 
funding  was  not  available  to  take  a  pre-experimental  trip. 
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Second,  some  of  the  computer  software  problems 
encountered  could  have  been  avoided  if  the  programmer  had 
some  experience  in  the  subject  area  of  the  CBT  lesson,  or  if 
the  subject  matter  expert  was  at  the  same  location.  Also, 
the  loading  problems  could  have  been  avoided  if  AFLMC  had 
tested  the  loading  of  the  program  on  hardware  comparable  to 
hardware  found  at  base  level  prior  to  forwarding  the  program 
to  the  authors. 

Summary 

Now  is  the  time  to  plan  for  the  future  of  CBT 
development  in  the  supply  career  field  and  in  the  Air  Force. 
This  study  has  clearly  shown  that  CBT  is  more  effective  than 
CDCs  for  training  new  supply  warehousemen,  AFSC  645X1.  When 
used  properly,  CBT  can  decrease  the  amount  of  time  spent  on 
training  while  simultaneously  increasing  learning  and  the 
ability  of  trainees  to  perform  the  tasks  for  which  they  have 
been  trained. 

The  Air  Force  supply  community  cannot  accept  the 
current  situation  of  CBT  development  because  the  future  will 
present  an  ever  increasing  demand  for  better,  more  effective 
training  techniques.  A  plan  needs  to  be  developed  now,  and 
the  resources  must  be  programmed  to  ensure  that  the  capacity 
exists  to  handle  the  future  demand. 

The  senior  leadership  within  the  Air  Force  must  be  made 
aware  of  the  tremendous  positive  impact  which  CBT  can  have 
on  the  workforce,  and  should  act  swiftly  to  develop  policy 
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and  guidance.  The  sooner  policy  is  established,  the  sooner 
the  Air  Force  can  exploit  the  existing  technology  and 
develop  training  systems  which  improve  the  capacity  of  our 
workforce  to  learn,  as  well  as  their  capacity  to  perform. 
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Appendix  Ai  Ar.tic.ls  sm  Air  Zones  CBT  Development 


THE  WAREHOUSE  INSTRUCTIONAL  WIZARD: 

THE  APPLICATION  OF  COGNITIVE  LEARNING  THEORIES  TO  COMPUTER- 

BASED  TRAINING 

Lt  Col  Tim  0.  Peterson,  Ph.  D. 

Air  Force  Logistics  Management  Center 
Montgomery,  Alabama 

Abstract 

This  paper  examines  the  application  of  cognitive 
science  findings  to  an  United  States  Air  Force  technical 
training  class  using  computer-based  training  (CBT).  The  CBT 
being  designed  and  developed  is  for  the  material  storage  and 
distribution  (MS&D)  career  field.  Initial  technical 
training  for  al  MS&D  specialists  will  be  conducted  through 
the  use  of  these  computer-based  training  modules  and  on-the- 
job  experience. 


Introduction 

Performance  occurs  when  an  individual  translates 
internally  stored  knowledge  into  specific  behaviors  of 
cognitive  processes.  For  example,  an  aircraft  mechanic 
demonstrates  the  application  of  stored  knowledge  to  a 
specific  behavior  when  (s)he  repairs  a  malfunctioning 
leading  edge  flap  on  a  fighter  aircraft.  Likewise,  when  a 
computer  programmer  conceptualizes  the  design  for  a  new 
inventory  control  program,  (s)he  is  demonstrating  the 
application  of  stored  knowledge  to  a  cognitive  process. 
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To  perform  any  mental  or  physical  task  requires  stored 
knowledge .  Stored  knowledge  is  acquired  through  the 
learning  process.  Learning  is  defined  as  the  acquisition  of 
knowledge  which  causes  a  relatively  permanent  change  in 
specific  behaviors  or  cognitive  processes.  As  Norman  (1982) 
points  out,  learning  is  more  than  just  remembering.  It  is 
also  the  ability  to  perform  some  task  with  skill.  Learning 
then  is  both  purposeful  remembering  and  skillful 
performance.  Learning  can  occur  in  many  different  wayB. 

For  example,  learning  can  occur  through  trial  and  error, 
imitation,  or  a  conscious  effort  to  master  some  body  of 
knowledge.  The  latter  form  of  learning,  when  intended  to 
provide  domain-specific  knowledge,  is  referred  to  as 
technical  training. 

Currently,  most  of  the  technical  training  conducted  in 
the  United  States  Air  Force  is  rooted  in  the  behavioral ist 
traditions  of  B.  F.  Skinner  (1971).  The  central  theme  of 
Skinner's  work  is  that  human  behaviors  are  controlled  by  the 
immediate  consequences  following  those  behaviors.  So 
behaviors  which  are  immediately  reinforced  will  be  repeated, 
and  behaviors  which  are  not  immediately  reinforced  will  not 
be  repeated.  The  application  of  Skinner's  operant  condi¬ 
tioning  principles  through  the  use  of  programmed  instruction 
has  consistently  shown  a  reduction  in  the  amount  of  time 
required  to  train  a  specific  technical  skill  (Mayo,  1969). 
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On  the  other  hand,  the  programmed  instruction  method 


does  not  seem  to  improve  the  acquisition  of  knowledge  or  the 
retention  of  knowledge  over  time  (Wexley  &  Latham,  1981). 

One  explanation  for  these  results  is  that  when  educators 
operationalize  the  operant  conditioning  principles,  the 
trainee  generally  is  relegated  to  the  role  of  passive 
receiver  and  transmitter.  Another  explanation  is  that  in  an 
effort  to  construct  strong  learning-knowledge-reward 
relationships,  trainers  simplify  the  learning  so  much  that 
the  learning  becomes  trivial.  The  insignificance  of  the 
learning  may  also  explain  why  training  times  are  always 
lower  using  programmed  instruction  techniques.  The  end 
results  are  these:  1)  training  becomes  linear  and 
sequential  in  nature,  2)  seldom  is  the  training  internalized 
or  integrated  with  other  knowledge,  and  3)  the  learning 
which  does  take  place  is  generally  only  measurable  by  paper 
and  pencil  tests  rather  than  actual  performance  of  the 
specific  task. 

With  the  emergence  of  cognitive  psychology  as  a  science 
over  the  past  thirty  years,  the  role  of  the  trainee  as  a 
passive  receiver  of  information  has  changed.  More  and  more 
cognitive  scientists  (Anderson,  1980;  Anderson,  1981; 

Lindsay  &  Norman,  1977  Norman,  1982)  are  arguing 
convincingly  that  learners  are  active  information 
processors,  deciding  what  information  is  important,  how  to 
organize  that  information,  how  to  store  that  information 
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internally,  and  how  (or  even  if)  to  use  that  information  in 
different  situations.  The  research  on  cognitive  information 
processing  has  led  to  new  learning  theories  and  methods. 

New  cognitive  learning  theories  promote  the  concept  of 
cognitive  model  development.  A  cognitive  model  for  a 
specific  task  is  an  individual's  mental  representation  of 
the  task.  In  other  words,  this  mental  model  consists  of 
specific  facts  or  principles  about  the  task  (declarative 
knowledge),  how  the  task  is  best  accomplished  (procedural 
knowledge),  and  information  about  how  this  task  fits  into 
other  tasks  (system  knowledge).  All  of  this  information  is 
integrated  into  an  organized  structure  often  referred  to  as 
a  schematic  net.  People  use  these  cognitive  models  to 
interpret  environmental  cues  and  to  decide  how  to  respond  to 
these- cues.  When  a  cognitive  model  is  tightly  integrated 
for  a  domain-specific  task,  the  individual  is  said  to  have 
compiled  knowledge.  Generally,  other  people  refer  to  this 
person  as  an  expert. 

The  complexity  that  cognitive  models  add  to  training 
domain-specific  technical  tasks  is  enormous.  For  that 
reason,  few  practical  applications  of  cognitive  learning 
theories  have  emerged.  Yet  the  benefits  which  can  be 
realized  are  alBo  enormous.  On  the  other  hand,  to  institute 
this  form  of  training  in  a  classroom  environment  would  mean 
substantially  extending  the  length  of  most  resident  courses 
(Bourne,  Dominowski,  Loftus,  &  Healy,  1986).  The  desire  of 


most  organizations  (including  the  Air  Force)  to  reduce 
classroom  time  and  move  the  training  closer  to  the  actual 
work  environment,  makes  the  use  of  cognitive  learning 
theories  very  unlikely. 

However,  by  combining  the  new  cognitive  learning 
theories  with  computer-based  technology  it  is  possible  to 
improve  the  learning  and  to  reduce  training  time 
simultaneously.  Ferguson  (1989)  refers  to  this  integration 
of  new  instructional  technology  and  cognitive  learning 
theories  as  a  student-centered  approach  versus  an 
instructor-centered  approach  to  technical  training.  In 
addition,  the  ability  to  export  computer-based  courseware  to 
the  actual  work  site  has  the  potential  of  reducing  classroom 
time  and  of  moving  the  training  directly  into  the  work 
environment.  While  these  benefits  seem  possible,  few 
applications  have  yet  emerged. 

This  is  the  central  issue  of  this  manuscript:  the 
implications  of  applying  cognitive  learning  theories  to 
computer-based  training  for  a  specific  technical  skill. 
Specifically,  the  technical  skill  under  development  is  the 
material,  storage  and  distribution  specialty  (i.e. 
warehousing  skills).  In  the  remainder  of  this  manuscript,  I 
will  discuss  cognitive  learning  principles  as  they  apply  to 
developing  a  mental  model  for  the  design  and  development  of 
computer-based  training  for  the  warehousing  task.  Specific 
cognitive  learning  theories  will  be  discussed  as  they 
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apply  to  tutoring,  practice,  and  mastery  for  the  warehousing 
task. 


Mental  Modela 

A  mental  model  for  any  task  or  activity  is  an 
individual's  conceptualization  of  the  work  domain 
(environmental  situation),  the  applicable  task  knowledge 
(declarative  knowledge),  and  how  to  apply  that  knowledge  to 
the  specific  task  (procedural  knowledge).  According  to 
Gott,  Bennett,  and  Gillet  (1986),  mental  models  serve  as 
road  maps  for  problem  solving  activities.  These  models 
influence  what  information  is  acquired  from  a  problem¬ 
solving  event,  how  problems  are  represented  in  an 
individual's  memory,  and  how  new  information  about  a 
specific  problem  is  organized  and  accessed  later  when  a 
similar  work  Betting  presents  itself.  Rummelhart  and  Norman 
(1981)  found  that  in  learning  problem-solving  skills,  the 
majority  of  errors  can  be  traced  to  faulty  mental  models.  A 
good  mental  model  allows  an  individual  to  predict  the 
effects  of  his/her  actions.  Norman  (1988)  argues  that 
without  a  good  mental  model  we  operate  by  rote  memory  -  only 
doing  what  we  are  told  to  do,  but  not  understanding  why  or 
being  able  to  explain  to  others  the  effects  of  our  efforts. 
Norman  says  working  without  a  mental  model  works  fine  as 
long  as  nothing  goes  wrong;  but  when  something  does  go  wrong 
or  when  we  are  faced  with  a  novel  situation,  we  are  stymied 
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because  we  do  not  have  a  deeper  mental  model  for  the  given 
task. 

While  mental  models  serve  a  critical  function  in  an 
individual's  development  of  technical  competency,  only 
recently  have  learning  theorists  started  to  empirically 
study  how  these  models  are  developed.  For  example,  Gott, 
Bennet,  &  Gil let  (1986)  found  that  greater  success  in 
converging  on  the  source  of  electronic  system  malfunction 
was  partially  explained  by  the  superior  mental  models 
possessed  by  the  skilled  electronics  technicians  versus  the 
less-skilled  technicians.  This  research  confirms  the 
importance  of  developing  mental  models  as  part  of  the 
technical  training.  Gagne  and  Classner  (1987)  suggest  that 
one  tactic  for  doing  this  is  to  attach  the  new  mental  model 
to  everyday  situations  or  similar  phenomena  the  trainee  has 
already  experienced. 

In  the  material,  storage  and  distribution  course,  we 
applied  this  principle  by  creating  two  separate  lessons. 

The  first  was  a  lesson  on  the  overall  Air  Force  Supply 
System.  In  this  lesson,  we  relate  Supply  to  Sears,  Wal- 
Mart,  7-11  Stores,  and  Texaco.  Throughout  this  lesson,  the 
MS&D  trainee  is  provided  familiar  mental  models  and  then 
those  models  are  related  to  the  Air  Force  Supply  System. 

The  objective  here  is  not  to  introduce  the  trainee  to  new 
terminology  or  specific  procedures,  but  to  develop  the 
skeleton  of  a  mental  model  of  the  supply  system.  In  the 
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second  lesson,  we  focus  in  on  the  material,  storage  and 
distribution  function.  In  this  lesson,  we  use  examples  from 
Sears  again,  but  also  mini-storage  facilities,  K-Mart 
distribution  centers,  and  delivery  services.  Upon 
completion  of  these  two  lessons,  the  MS&D  trainees  have  a 
reasonably  good  mental  framework  upon  which  to  build.  Mager 
(1988)  describes  this  as  providing  the  student  with  the  big 
picture  or  a  mental  framework  upon  which  to  store  other 
knowledge.  Based  on  this  framework,  Mager  (1988)  argues 
that  performance  objectives  can  be  generated  for  specific 
tasks  the  trainee  needs  to  perform. 

Performance  Hierarchies 

Performance  objectives  are  word  descriptions  of  the 
performance  a  trainee  needs  to  exhibit  before  (s)he  is 
considered  competent.  In  the  MS&D  course,  each  task  was 
defined  by  a  performance  objective.  For  example,  being  able 
to  store  property  with  an  assigned  warehouse  location,  being 
able  to  process  organizational  refusals,  being  able  to 
perform  a  bench  stock  inventory,  being  able  to  input 
receiving  documents  to  the  base  supply  computer  system  are 
all  examples  of  performance  objectives.  These  objectives 
each  identify  the  intended  result  of  the  lesson,  rather  than 
the  process  of  the  Instruction.  In  addition,  each  of  these 
performance  objectives  focuses  on  what  the  trainee  must  do 
to  demonstrate  competence.  From  the  performance  objectives, 
performance  hierarchies  are  created.  A  performance 
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hierarchy  is  a  graphical  representation  of  the  knowledge 
clusters  needed  to  perform  the  stated  objective.  Mager 
(1984)  argues  that  performance  hierarchies  help  the 
instructional  developer  answer  the  question  "what  does  the 
trainee  need  to  know  to  perform  this  task?"  Performance 
hierarchies  also  provide  a  graphical  image  of  the 
relationship  between  different  pieces  of  knowledge.  This 
information  aids  the  instructional  developer  in  determining 
the  sequencing  of  task  information.  For  every  performance 
objective  in  the  MS&D  course,  a  performance  hierarchy  was 
created.  One  additional  benefit  of  these  hierarchies  was 
that  17  repeating  knowledge  clusters  were  identified  across 
90  different  performance  objectives.  These  seventeen 
clusters  were  combined  into  separate  "principles  of 
warehousing"  modules.  By  reclustering  this  knowledge  into  a 
separate  module,  this  material  could  be  taught  as 
propaedeutic  skills  to  all  MS&D  specialists  early  in  their 
training  program.  These  learning  theory  principles  were 
applied  to  all  lessons  to  ensure  that  all  lessons  were 
structures  around  optimal  mental  model  development.  The 
performance  hierarchies,  once  created,  were  then  used  to 
design  and  develop  the  computer-based  tutorial  for  each 
hierarchy. 


Tutorial 

Each  performance  objective  has  a  tutorial  which  is 
self-paced  and  is  individualized  to  the  trainees  based  on 
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their  responses  to  memory  check  items  throughout  the 
tutorial.  Criswell  (1989)  calls  this  form  of  tutorial  a 
"branching  program"  which  adapts  to  the  individual  trainee 
as  the  material  is  presented.  In  a  branching  tutorial,  time 
and  content  vary  according  to  the  trainee's  needs.  This  is 
consistent  with  Reynolds  (1987),  who  says  a  good  computer- 
based  tutorial  lesson  advances  just  like  a  good  human  tutor 
would,  based  on  the  interaction  between  the  tutor  and  the 
student.  Fitts  and  Posner  (1967)  refer  to  this  as  the 
cognitive  stage  of  learning  during  which  a  trainee  forms  a 
mental  representation  of  how  a  skill  is  performed. 

In  addition,  the  self-paced  and  individualize  nature  of 
the  computer-based  lesson  is  consistent  with  what  we  know 
about  adult  learners.  Knowles  (1979)  reports  that  adult 
learners  vary  in  their  rate  of  learning,  that  they  enter  the 
learning  situation  with  varying  degrees  of  experience,  and 
that  they  have  different  learning  styles  and  preferences. 
With  these  factors  in  mind,  it  seems  that  the  optimal 
learning  strategy  would  be  to  have  the  training  system  adapt 
to  the  learner's  past  experience,  knowledge,  and  current 
needs.  In  an  effort  to  integrate  these  concepts  into  the 
MS&D  courseware,  another  form  of  adaptive  training  is 
capitalized  on.  When  knowledge  from  one  of  the  earlier 
lessons  was  needed  in  a  later  lesson,  the  trainee  was  given 
a  short  pre-test  (consisting  of  three  questions)  on  the 
principle  in  question.  Based  on  answer  analysis  of  the  pre- 
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test,  the  current  computer-based  lesson  determines  the 
amount  of  refresher  training  necessary.  The  refresher 
training  could  consist  of  no  additional  training  at  all  if 
the  trainee  got  all  of  the  questions  right.  On  the  other 
hand,  if  the  trainee  missed  only  the  hard  question,  the 
refresher  training  may  be  a  one  screen  summary  of  the 
information.  Complete  retraining  of  the  principle  might 
automatically  occur  given  that  the  trainee  missed  all  three 
questions. 

All  of  the  tutorial  lessons  adapt  according  to  the 
responses  of  the  trainees.  Based  on  the  response  of  a 
trainee  to  a  memory  check  item,  the  computer-based  lesson 
analyzes  the  response  and  provides  the  trainee  with  either 
positive  feedback  or  tailored  remediation  for  each  of  the 
other .responses .  The  positive  feedback  not  only  tells  the 
trainee  that  the  correct  answer  has  been  chosen,  but  also 
provides  expanded  information  on  the  topic.  These  positive 
feedback  blocks  are  also  color  coded  on  the  computer  screen 
to  send  a  positive  visual  display  to  the  trainee.  The 
remediation  feedback  blocks  are  each  tailored  to  the 
specific  response  item  chosen.  In  this  way,  the  specific 
error  made  is  remediated  and  the  correct  information 
reinforced.  Just  as  the  positive  feedback  blocks  are  color 
coded,  so  are  the  remediation  blocks,  but  with  a  different 
color. 
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Each  tutorial  lesson  also  starts  with  information  on 


how  this  specific  performance  objective  fits  into  the  mental 
framework  created  in  the  first  two  lessons  of  this  course. 

It  also  explains  why  this  material  is  important  to 
understanding  the  performance  task  at  hand.  MacLachlan 
(1986)  reports  that  explaining  the  benefit  of  a  tutorial 
lesson  increases  a  trainee's  attention.  By  increasing  a 
trainee's  attention  level,  we  also  increase  the  likelihood 
that  more  information  will  be  retained  by  the  trainee 
(Criswell,  1989). 

To  aid  the  trainee  in  organizing  the  information,  all 
learning  clusters  or  segments  within  a  lesson  start  with  an 
advance  organizer.  An  advanced  organizer  aids  a  trainee 
like  a  road  map  aids  a  traveler.  It  provides  an  overview  of 
how  this  information  relates  to  the  overall  performance 
objective.  It  can  also  show  how  past  knowledge  clusters  are 
related  to  this  objective  and  how  this  task  fits  into  the 
larger  performance  framework.  In  many  ways,  an  advanced 
organizer  helps  prepare  the  trainee  for  the  learning 
process.  By  providing  an  advanced  organizer,  the  trainee 
has  a  knowledge  skeleton  upon  which  to  build.  Clark  (1987) 
argues  that  providing  an  advanced  organizer  increases  the 
attention  level  of  the  trainee,  provides  the  trainee  with  a 
structure  upon  which  to  hang  other  details,  and  aids  the 
trainee  with  the  management  of  the  learning  process. 
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In  addition,  each  learning  cluster  is  concluded  with  a 
summary  of  the  essential  information  in  that  segment.  Once 
again,  Clark  (1987)  argues  that  the  summary  provides  the 
trainee  the  opportunity  to  reexamine  his/her  own  knowledge 
structure  to  ensure  that  the  essential  information  has  been 
stored. 


Practice 

In  addition  to  the  tutorial  for  each  performance 
objective,  each  objective  also  has  a  practice  component 
which  follows  the  tutorial.  Salisbury  (1988)  posits  that, 
no  matter  how  effective  the  initial  tutorial  is,  unless  the 
trainee  is  allowed  sufficient  time  to  practice,  (s)he  may 
never  demonstrate  mastery  of  the  task.  The  practice 
component  allows  the  trainee  to  apply  the  knowledge  (s)he 
just  learned  in  the  tutorial.  Fitts  and  Posner  (1967)  call 
this  "the  associative  stage  of  learning. "  Here  the  trainee 
attempts  to  perform  the  task  using  the  knowledge  acquired  in 
the  cognitive  stage  (tutorial).  Usually  errors  or 
misunderstandings  are  identified  during  the  early  practice 
sessions.  With  remediation  and  more  practice,  execution  of 
the  skill  is  improved  until  the  trainee  performs  the  task 
effortlessly  and  without  assistance  from  the  computer.  In 
some  ways,  computer-based  practice  is  like  walking  a 
tightrope  with  a  safety  net. 
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Each  practice  session  is  scenario-based.  The  trainee 
is  presented  with  a  specific  work  setting  and  then  is  asked 
to  respond  to  some  form  of  inquiry.  Salisbury  (1988)  calls 
this  type  of  practice  "drills  for  intellectual  skills."  In 
this  type  of  practice,  work  problems  are  presented  to  the 
trainee  and  the  trainee  is  asked  to  solve  the  problem  using 
prior  knowledge.  For  example,  a  scenario  might  read  as 
follows: 

You  have  Just  returned  from  lunch  and  your  supervisor 
asks  you  to  store  the  property  which  was  brought  over 
from  the  Receiving  Section.  You  go  out  to  the  holding 
area  and  find  three  items  with  attached  Notices  to  Stock 
Documents.  The  first  item's  Notice  to  Stock  Document  is 
shown  to  you  on  the  screen.  Will  you  store  this  piece  of 
property  in  your  warehouse? 

Based  on  the  answers  to  this  scenario,  the  computer- 
based  practice  component  will  determine  if  and  how  much 
additional  training  is  necessary  on  this  concept.  Based  on 
the  answer  analysis,  the  practice  component  will  either 
route  the  trainee  to  additional  training  on  this  concept  or 
advance  to  the  next  scenario.  The  practice  session  can  be 
repeated  as  often  as  the  trainee  wishes,  allowing  the 
trainee  to  fine  tune  the  skill  and  develop  confidence  in 
his/her  ability. 


Mastery 

If  the  practice  component  is  like  walking  a  tightrope 
with  a  net,  then  mastery  is  like  performing  the  task  without 
a  net.  Sullivan  and  Elenburg  (1988)  call  this  type  of 
evaluation  "performance  testing."  Performance  tests  are 
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based  on  the  performance  objective,  just  as  the  tutorial 
and  practice  components  are.  The  performance  test  simulates 
the  actual  work  environment  and  the  work  situation  that  will 
be  experienced  in  that  setting.  It  provides  work  situations 
to  the  trainee.  Based  on  the  trainee's  response  to  these 
situations,  the  training  system  evaluates  the  trainee's 
ability  to  correctly  perform  the  necessary  motor  or 
cognitive  skill  to  successfully  complete  the  task.  These 
work  setting  simulations  provide  greater  realism  in  the 
testing  environment.  The  simulation  allow  the  trainee  to 
follow  a  line  of  reasoning  to  its  logical  conclusion.  It 
should  be  pointed  out  that  the  reasoning  may  have  been 
flawed  early  in  the  simulation,  but  the  computer  simulation 
will  allow  the  trainee  to  follow  that  line  of  reasoning  to  a 
logical  conclusion.  By  analyzing  the  individual  responses 
throughout  the  simulation,  the  computer-based  performance 
test  provides  very  valuable  information  on  where  the  flaw  is 
in  the  knowledge  base  and  what  remediation  is  necessary  to 
repair  the  knowledge  base . 

Computer-based  performance  tests  also  allow  for 
adaptive  testing  techniques  (Ward,  1984).  Each  response  can 
be  analyzed  and,  based  on  past  responses,  the  most 
appropriate  next  question  can  be  asked.  For  example,  in  the 
MS&D  mastery  component,  each  trainees  response  is  examined; 
if  the  response  is  determined  to  be  incorrect,  another 
scenario  based  on  the  same  information  is  presented  to  the 
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trainee.  If  the  trainee  performs  correctly  in  the  second 
scenario,  (s)he  is  allowed  to  pass  through  the  gateway  for 
that  knowledge  cluster.  On  the  other  hand,  if  the  trainee 
performed  incorrectly  on  the  second  effort,  the  computer- 
based  master  component  notifies  the  trainee  what  specific 
knowledge  is  deficient  and  in  need  of  remediation.  While 
there  are  over  thirty  different  scenarios  in  the  mastery,  a 
skilled  trainee  will  complete  mastery  by  answering  only  ten 
scenarios.  The  other  scenarios  are  needed  to  refine  and 
identify  specific  knowledge  flaws. 

Summflyv 

Based  on  accepted  cognitive  learning  principles,  we 
know  that  learners  are  not  passive  receptors  of  information. 
They  are  active  information  processors  attending  to 
information  which  the  learner  believes  will  assist  him/her 
in  accomplishing  a  specific  task.  With  this  knowledge,  we 
can  provide  learners  with  mental  models  which  will  assist 
the  learner  in  the  learning  process.  We  know  that  practice 
of  a  skill  greatly  influences  late  performance.  Through  the 
use  of  answer  analysis  and  remediation  techniques,  a 
learner's  skill  level  can  be  improved  through  computer-based 
practice.  Finally,  by  combining  microcomputer  power  with 
adaptive  testing  techniques,  we  are  able  to  more  accurately 
measure  an  individual's  learning  and  the  specific  flaws  in 
that  learning. 
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Step  1.  Get  AQE  scores  in  general  category  for  test 
subjects  from  CBPO. 

Step  2.  Determine  treatment  composition. 

a.  Rank  students  by  score  on 
AQE  test  (highest  to  lowest). 

b.  Divide  scores  into  quartiles.  If  the 
number  of  students  is  evenly  divisible  by 
4,  then  all  quartiles  will  be  of  equal 
size.  If  the  number  of  students  divided 
by  4  leaves  a  remainder,  then  assign  the 
extra  student (s): 


Step  3.  Use  the  random  number  table  to  assign  the  test 
subjects  in  each  quartile  to  a  treatment. 


a.  Choose  a  stream  of  2  digit  random 
numbers. 


b.  Assign  test  subjects  to  the  treatments  in 
this  order:  CBT,  CDC,  Control  group 

c.  Breakpoints  are  determined  by  dividing 
the  total  number  of  students  in  the 
quartile  by  100.  For  example,  if  there 
are  4  students  in  the  quartile,  then 
there  will  be  4  breakpoints,  00-24,  25- 
49,  50-74,  75-99. 

If  the  first  random  number  is  24,  then 
student  number  1  will  be  assigned  to  the 
CBT  treatment.  If  the  second  random 
number  is  between  00-24,  then  you  must 
skip  this  random  number  and  pick  again 
(because  00-24  represents  student  1  who 
has  already  been  assigned  to  a 
( treatment ) . 

d.  Repeat  process  for  students  in  other  2 
quartiles . 


Appendix  £:  Prepared  Introductions  tor  Treatments 


A.  CBT  Treatment 


Good  afternoon,  my  name  is  Capt  _ ,  and  I 

would  like  to  welcome  all  of  you  to  Lowry  AFB.  Today  you 
will  begin  your  supply  training  by  using  the  personal 
computer  to  take  a  supply  lesson.  Once  you  have  finished 
the  lesson,  you  will  take  a  test.  I  am  going  to  give  you 
some  specific  information  which  you  must  understand  before 
we  begin,  and  then  I'll  answer  any  questions  you  may  have. 

It  is  very  important  that  you  pay  attention  while  I  am 
giving  you  these  instructions.  If  you  don't  understand  any 
of  the  instructions,  please  tell  me  before  we  begin  the 
training  lesson. 

OVKHD-Does  your  computer  show  the  '‘C:\QUEST"  prompt? 

-There  is  no  time  limit  for  you  to  complete  the 
training.  However,  you  must  complete  the  training 
material  before  you  can  take  a  break. 

-During  the  training,  only  one  person  is  permitted  to 
use  the  restroom  at  a  time. 

-If  you  finish  the  training  lesson  early,  you  must 
remain  seated  and  quiet  until  everyone  is  finished. 

-Also,  during  the  break,  you  are  not  allowed  to  discuss 
any  information  from  the  lesson  with  other  airmen. 
QVBHB-A  few  times  during  the  training,  you  will  have  the 

choice  to  continue  on,  review  material,  stop  and  save 
your  place,  or  stop  without  saving  your  place.  When 
you  reach  these  decision  points,  feel  free  to  continue 
or  review  material:  NEVER  stop  the  lesson  by  pressing 
"X"  or  "F10" . 

OVRHD- During  the  lesson,  if  you  see  a  screen  that  tells  you 
to  contact  your  trainer  or  your  supervisor  for  help, 
let  me  know  IMMEDIATELY.  I  will  have  to  adjust  your 
computer  so  you  can  continue. 

OVRHD-When  you  reach  the  end  of  the  training  material,  you 
will  answer  some  practice  questions  before  taking  the 
test.  After  answering  the  last  practice  question,  you 
will  see  a  green  box  on  your  screen  which  says, 

"Great,  you've  got  the  idea.  Now  you  are  ready  to 
take  the  test."  When  you  reach  this  point,  STOP  and 
raise  your  hand. 

I  don't  want  you  to  go  past  that  screen  until  everyone 
is  finished  with  the  training. 

-Before  we  begin  the  test,  you  will  be  given  a  15 
minute  break  to  relax,  use  the  restroom,  etc.  During 
the  break,  you  must  remain  in  the  break  area  (or 
whatever  location  is  assigned). 

-Some  of  you  may  know  a  lot  about  computers,  and  some 
of  you  probably  don't  know  anything.  If  you  are  not 
familiar  with  computers,  don't  worry.  Once  the 
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training  lesson  begins,  all  you  have  to  do  is  follow 
the  instructions  on  the  computer. 

-Before  we  begin,  let  me  remind  you  that  as  members  of 
the  United  States  Air  Force,  you  are  bound  by  a  code 
of  honor.  Therefore,  I  expect  all  of  you  to  follow  my 
instructions  carefully. 

»  -During  the  training,  let  me  know  if  you  are  having  any 

problems,  or  if  you  have  any  questions.  I  will  try  to 
answer  your  questions  as  best  as  I  can. 

Does  anyone  have  any  questions  before  we  begin? 

OVRHD-Please  type  the  word  "CBTEVL  and  hit  the  <enter>  key. 

QVRHD- Remember .  don't  press  "X"  or  "F10"  when  given  the 
choice,  and  when  you  see  the  other  two  statements, 

STOP  and  raise  your  hand. 

You  may  begin. 


B.  CDC  Treatment 


Good  afternoon,  my  name  is  Capt  _ ,  and  I  would 

like  to  welcome  all  of  you  to  Lowry  AFB.  Today  you  will 
begin  your  supply  training  by  taking  a  supply  lesson.  Once 
all  of  you  have  finished,  you  will  all  take  a  test. 

I  am  going  to  give  you  some  specific  information  which 
you  must  understand  before  we  begin,  and  then  I'll  answer 
any  questions  you  may  have.  It  is  very  important  that  you 
pay  attention  while  I  am  giving  you  these  instructions.  If 
you  don't  understand  any  of  the  instructions,  please  tell  me 
before  we  begin  the  training  module. 

-There  is  no  time  limit  for  you  to  complete  the 
training.  However,  you  must  complete  the  training 
material  before  you  can  take  a  break. 

-At  the  end  of  each  section  in  the  training  material, 
there  are  questions  which  I  encourage  you  to  answer 
before  you  continue  on.  Feel  free  to  write  in  these 
booklets . 

-During  the  training,  one  person  is  permitted  to  use 
the  restroom  at  a  time. 

-If  you  finish  the  training  material  early,  you  must 
remain  seated  and  quiet  until  everyone  is  finished. 

-After  the  training  module  is  finished,  you  will  be 
given  a  15  minute  break  to  relax,  use  the  restroom, 
etc.  During  the  break,  you  must  remain  in  the  break 
area  (or  whatever  location  is  assigned). 

-Also,  during  the  break,  you  are  not  allowed  to  discuss 
any  information  from  the  training  material  with  other 
airmen. 
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Does  anyone  have  any  questions  at  this  point? 

This  training  material  is  self-explanatory.  Read  the 
material,  and  I  encourage  you  to  answer  the  exercise 
questions  before  going  on.  The  answers  to  the  exercise 
questions  are  in  the  back  of  your  training  package. 

Before  we  begin,  let  me  remind  you  that  as  members  of 
the  United  States  Air  Force,  you  are  bound  by  a  code  of 
honor.  Therefore,  I  expect  all  of  you  to  follow  my 
instructions  carefully. 

During  the  training,  let  me  know  if  you  are  having  any 
problems,  or  if  you  have  any  questions.  I  will  try  to 
answer  your  questions  as  best  as  I  can. 

Does  anyone  have  any  questions?  You  may  begin. 


C.  Control  Group 


Good  afternoon,  my  name  is  Capt  _ ,  and  I  would 

like  to  welcome  all  of  you  to  Lowry  AFB.  Today  you  will  use 
a  personal  computer  to  take  a  test  to  see  how  much  knowledge 
you  have  of  some  basic  supply  operations. 

I  am  going  to  give  you  some  specific  information  which 
you  must  understand  before  we  begin,  and  then  I'll  answer 
any  questions  you  may  have.  It  is  very  important  that  you 
pay  attention  while  I  am  giving  you  the  instructions.  If 
you  don't  understand  any  of  the  instructions,  please  tell  me 
before  we  begin  the  test. 

-there  is  no  time  limit  on  this  test. 

-raise  your  hand  as  soon  as  you  are  finished  with  the 
test.  Afterward,  you  may  go  to  the  break  area. 

-Once  everyone  is  done,  I  will  bring  you  back  in  the 
room  for  further  instructions. 

-during  the  test  and  the  break,  you  are  not  allowed  to 
discuss  the  test  with  any  other  airmen. 

OVRHD-does  your  computer  screen  show  the  "C:\QUEST"  prompt? 

-there  are  13  questions  on  the  test. 

QVRHD-STOP .  raise  your  hand  when  you  reach  the  ”C:>"  prompt. 

Does  anyone  have  any  questions  at  this  point? 

This  test  is  self-explanatory.  Follow  the  instructions 
on  the  computer,  and  answer  the  questions  to  the  best 
of  your  ability. 

Before  we  begin,  let  me  remind  you  that  as  members  of 
the  United  States  Air  Force,  you  are  bound  by  a  code  of 
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honor.  Therefore,  I  expect  all  of  you  to  follow  my 
instructions  carefully. 

Turing  the  test,  let  me  know  if  you  are  having  any 
problems,  or  if  you  have  any  questions.  I  will  be 
happy  to  answer  your  questions  as  best  as  I  can. 

Does  anyone  have  any  questions  before  we  begin? 

Remember,  read  the  test  questions  carefully  and  answer 
them  as  well  as  you  can. 

OVRHD  Please  type  the  word  "PRETEST"  and  hit  the  <ENTER> 
key. 

You  may  begin. 
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Appendix  E:  Introduction  Task  Analysis 


You  are  assigned  to  the  Storage  and  Issue  section  of 
this  supply  squadron.  This  piece  of  property  was  delivered 
to  base  supply  today  by  a  truck.  The  people  in  the 
receiving  section  of  base  supply  in-processed  this  piece  of 
property  on  the  computer,  and  when  they  did,  this  piece  of 
paperwork  printed  out  on  their  printer.  Someone  from  the 
receiving  section  brought  this  piece  of  property  and  this 
paperwork  to  you.  Your  job  is  do  whatever  is  necessary  to 
properly  store  this  property  in  the  warehouse.  While  you 
are  actually  performing  this  task,  you  should  describe  to  me 
what  you  are  doing  so  that  I  can  understand  exactly  what  you 
are  doing. 
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Appendix  £Ll  Task  Evaluation  Checklist 
TASK  EVALUATION  CHECKLIST 


SUBJECT  (Name):  Start  Time: 

Completion  Time: 

Ask  the  airman  to  describe  the  steps  he/she  takes  to  store 
the  property.  Put  a  check  in  each  blank  space  when  the 
airman  correctly  completes  the  indicated  step  in  the  property 
location  procedure. 


Initial  Check  of  Property  and  Documentation 

1.  _  Is  a  MGT  Notice  or  a  DOR  attached  to  the  property? 

2.  _  Did  the  airman  determine  if  the  property  was 

forwarded  to  the  correct  warehouse? 

Finding  the  Warehouse  Location 


3.  _ 

_  Did  the  airman  determine 

from  the  MGT  Notice? 

the 

correct 

stockroom 

4.  _ 

_  Did  the  airman 

determine 

the 

correct 

bin  row? 

5.  _ 

; _  Did  the  airman 

bin  location? 

determine 

the 

correct 

horizontal 

6.  _ 

_  Did  the  airman 

location? 

determine 

the 

correct 

vertical  bin 

Storing  Er.opert.y  in  .the  Warehouse  Location 

7.  _  Did  the  airman  correctly  compare  the  Stock  Number 

and  Warehouse  Location  on  the  MGT  Notice  with  the 
Bin  Label  at  the  Warehouse  Location? 

8.  _  Did  the  airman  determine  if  the  quantity  of  the 

property  to  be  stored  agreed  with  the  quantity  on 
line  2  of  the  MGT  Notice? 

9.  _  Did  the  airman  compare  the  Stock  Number  on  the 

property  to  be  stored  with  the  Stock  Number  on  the 
property  in  location? 

10.  _  Did  the  airman  sign  and  date  Block  9  of  the  MGT 

Notice  and  distribute  the  document? 
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Appendix  £_l  CBT  Experimental  Sub.iect  Questionnaire 


Questionnaire 


NAME: 

This  questionnaire  is  designed  to  provide  us  with  important 
information  needed  for  our  study.  Please  write  your  name  in 
the  space  provided  above.  The  information  you  provide  will 
be  used  only  in  this  study  and  not  for  any  other  purposes  in 
accordance  with  the  USAF  Privacy  Act. 

Please  answer  the  following  questions  carefully.  Circle  only 
one  response  unless  otherwise  indicated. 


1.  Has  anyone  given  you  any  information  about  this  segment 
of  your  training  since  your  arrival  at  Lowry  AFB? 

a.  Yes 

b.  No 


2.  What  is  your  sex? 

a.  Male 

b.  Female 


3.  What  is  your  age? 

a.  Under  18 

b.  18  to  19 

c.  20  to  21 

d.  22  to  23 

e.  24  to  25 

f.  26  or  Older 


4.  What  is  your  highest  educational  level? 

a.  Non-high  school  graduate 

b.  High  school  graduate  or  equivalent 

c.  Less  than  two  years  of  college 

d.  Associates  degree  or  equivalent 

e.  Bachelors  degree 

f.  Masters  degree 
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5.  Do  you  have  any  experience  using  computers? 

a.  Yes 

b.  No 

If  you  answered  yes  to  the  above  question  continue  to  the 
next  question.  If  you  answered  no  to  the  above  question, 
turn  the  questionnaire  in  to  your  monitor. 


6.  How  long  have  you  been  using  computers? 

a.  6  months  or  less 

b.  More  than  6  months  -  Less  than  1  year 

c.  More  than  1  year  -  Less  than  2  years 

d.  More  than  2  years  -  Less  than  3  years 
f.  3  years  or  more 


7.  What  type  of  software  do  you  have  experience  with? 
(You  may  circle  more  than  one  answer) 

a.  Word  processing  software  (i.e.  Word  Perfect) 

b.  Spreadsheet  software  (i.e.  Lotus  1-2-3) 

c.  Database  software  (i.e.  dBase  III) 

d.  Graphics  software  (i.e.  Harvard  Graphics) 

e .  Game  software 


8.  Do  you  have  any  programming  experience? 

a.  Yes 

b.  No 

If  you  answered  yes  continue  to  the  next  question.  If  you 
answered  no,  you  are  finished  with  the  questionnaire.  Please 
turn  the  questionnaire  into  your  monitor. 


9.  What  programming  language  are  you  familiar  with? 
(You  may  circle  more  than  one  answer) 


a.  FORTRAN 

b.  BASIC 

c .  COBOL 

d.  DOS 

e .  PASCAL 

f.  dBASE 

g.  Other 

Please  turn  your  questionnaire  in  to  your  monitor. 
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ONE  WAY  AOV 

SAMPLE  GROUP 

VARIABLE  MEAN  SIZE  VARIANCE 


SCOREEXP  58.98  49  231.4 
SCON206  59.31  206  145.2 
TOTAL  59 . 24  255 


SOURCE 

DF 

SS 

MS 

F 

P 

BETWEEN 

1 

4.213 

4.213 

0.03 

0.8718 

WITHIN 

253 

4 . 087E+04 

161.5 

TOTAL 

254 

4 . 087E+04 

CHI  SQ  DF  P 

BARTLETT'S  TEST  OF  -  -  - 

EQUAL  VARIANCES  4.60  1  0.0319 

COCHRAN'S  Q  0.6145 

LARGEST  VAR  /  SMALLEST  VAR  1.594 

COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS  -4.680E-01 
EFFECTIVE  CELL  SIZE  336.2 

CASES  INCLUDED  255  MISSING  CASES  157 


RANK  SUM  TWO  SAMPLE  TEST  FOR  SCOREEXP  VS  SCON206 

SAMPLE  AVERAGE 

VARIABLE  RANK  SUM  SIZE  U  STAT  RANK 

SCOREEXP  6 . 104E+03  49  4.879E+03  124.6 

SCON206  2 . 654E+04  206  5.214E+03  128.8 

TOTAL  3 . 264E+04  255 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  0.360 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.7189 

CASES  INCLUDED  255  MISSING  CASES  157 


SCOREEXP-AQE  SCORES  FOR  645X0  EXPERIMENTAL  SUBJECTS 
SCON206-AQE  SCORES  FOR  RANDOM  SAMPLE  OF  645X1  PERSONNEL 
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KRUSKAL-WALLIS  ONEWAY  NONPARAMETRIC  AOV 


VARIABLE 

MEAN 

RANK 

SAMPLE 

SIZE 

TXTAQE 

27.0 

17 

CBTAQE 

24.8 

17 

CONAQE 

22.9 

15 

TOTAL 

25.0 

49 

KRUSKAL-WALLIS  STATISTIC  0.6624 

P  VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.7181 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F 

P 

BETWEEN 

2 

134.8 

67.40 

0.32 

0.7264 

WITHIN 

46 

9 . 633E+03 

209.4 

TOTAL 

48 

9 . 768E+03 

CASES  INCLUDED  49  MISSING  CASES  2 


TXTAQE-AQE  SCORES  FOR  THE  TEXT  GROUP 
CBTAQE-AQE  SCORES  FOR  THE  CBT  GROUP 
CONAQE-AQE  SCORES  FOR  THE  CONTROL  GROUP 
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KRUSKAL-WALLI S  ONEWAY  NONPARAMETRIC  AOV 


VARIABLE 

MEAN 

RANK 

SAMPLE 

SIZE 

TXTTST 

25.4 

17 

CBTTST 

38.4 

17 

CONTST 

9.4 

15 

TOTAL 

25.0 

49 

KRUSKAL-WALLI S  STATISTIC  33.2130 

P  VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.0000 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F 

P 

BETWEEN 

2 

6 . 7 10E+03 

3 . 355E+03 

51.66 

0.0000 

WITHIN 

46 

2 . 988E+03 

64.95 

TOTAL 

48 

9 . 698E+03 

CASES  INCLUDED  49  MISSING  CASES  2 


RANK  SUM  TWO  SAMPLE  TEST  FOR  TXTTST  VS  CBTTST 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

TXTTST 

197.0 

17 

44.00 

11.6 

CBTTST 

398.0 

17 

245.0 

23.4 

TOTAL 

595.0 

34 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  3.444 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.0006 


CASES  INCLUDED  34  MISSING  CASES  0 


TXTTST-TEST  SCORES 
CBTTST -TEST  SCORES 
CONTST-TEST  SCORES 


FOR  THE  TEXT  GROUP 
FOR  THE  CBT  GROUP 
FOR  THE  CONTROL  GROUP 
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RANK  SUM  TWO  SAMPLE  TEST  FOR  TXTTST  VS  CONTST 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

TXTTST 

387.0 

17 

234.0 

22.8 

CONTST 

141.0 

15 

21.00 

9.4 

TOTAL 

528.0 

32 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  4.003 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.0001 

CASES  INCLUDED  32  MISSING  CASES  2 


RANK  SUM  TWO  SAMPLE  TEST  FOR  CBTTST  VS  CONTST 


VARIABLE 

RANK 

SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

CBTTST 

408.0 

17 

255.0 

24.0 

CONTST 

120.0 

15 

0.000 

8.0 

TOTAL 

528.0 

32 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION 

4.796 

TWO  TAILED 

P  VALUE 

FOR 

NORMAL  APPROXIMATION 

0.0000 

CASES  INCLUDED  32  MISSING  CASES  2 
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KRUSKAL-WALLIS  ONEWAY  NONPARAMETRIC  AOV 


VARIABLE 

MEAN 

RANK 

SAMPLE 

SIZE 

TXTTSK 

25.0 

17 

CBTTSK 

38.1 

17 

CONTSK 

10.2 

15 

TOTAL 

25.0 

49 

KRUSKAL-WALLIS  STATISTIC  32.0570 

P  VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.0000 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F 

P 

BETWEEN 

2 

6. 185E+03 

3 . 092E+03 

46.25 

0.0000 

WITHIN 

46 

3. 076E+03 

66.87 

TOTAL 

48 

9 . 260E+03 

CASES  INCLUDED  49  MISSING  CASES  2 


RANK  SUM  TWO  SAMPLE  TEST  FOR  TXTTSK  VS  CBTTSK 


SAMPLE 

AVERAGE 

VARIABLE 

RANK  SUM 

SIZE 

U  STAT 

RANK 

TXTTSK 

203.0 

17 

50.00 

11.9 

CBTTSK 

392.0 

17 

239.0 

23.1 

TOTAL 

595.0 

34 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  3.238 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.0012 

CASES  INCLUDED  34  MISSING  CASES  0 


TXTTSK-TASK  SCORES  FOR  THE  TEXT  GROUP 
CBTTSK-TASK  SCORES  FOR  THE  CBT  GROUP 
CONTSK-TASK  SCORES  FOR  THE  CONTROL  GROUP 
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RANK  SUM  TWO  SAMPLE  TEST  FOR  TXTTSK  VS  CONTSK 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

TXTTSK 

375.0 

17 

222.0 

22.1 

CONTSK 

153.0 

15 

33.00 

10.2 

TOTAL 

528.0 

32 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  3.550 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.0004 

CASES  INCLUDED  32  MISSING  CASES  2 


RANK  SUM 

TWO  SAMPLE  TEST 

FOR  CBTTSK  VS  CONTSK 

SAMPLE 

AVERAGE 

VARIABLE 

RANK  SUM 

SIZE 

U  STAT 

RANK 

CBTTSK 

408.0 

17 

255.0 

24.0 

CONTSK 

120.0 

15 

0.000 

8.0 

TOTAL 

528.0 

32 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  4.796 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.0000 

CASES  INCLUDED  32  MISSING  CASES  2 


* 
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RANK  SUM  TWO  SAMPLE  TEST  FOR  TXTRT  VS  CBTTRT 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

TXTRT 

393.5 

17 

240.5 

23. 1 

CBTTRT 

201.5 

17 

48.50 

11.9 

TOTAL 

595.0 

34 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  3.289 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  O.OOIO 

CASES  INCLUDED  34  MISSING  CASES  0 


TXTRT-TEXT  GROUP  TRAINING  TIME 
CBTTRT-CBT  GROUP  TRAINING  TIME 
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RANK  SUM  TWO  SAMPLE  TEST  FOR  HSED  VS  COLED 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

HSED 

91.00 

11 

25.00 

8.3 

CO  LED 

62.00 

6 

41.00 

10.3 

TOTAL 

153.0 

17 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  0.754 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.4510 

CASES  INCLUDED  17  MISSING  CASES  5 


HSED- TEST  SCORES  FOR  HIGH  SCHOOL  EDUCATION  IN  CBT  GROUP 
COLED-TEST  SCORES  FOR  SOME  COLLEGE  EDUCATION  OR  ABOVE  IN  CBT 
GROUP 


RANK  SUM  TWO  SAMPLE  TEST  FOR  HSEDTK  VS  COLEDTK 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

HSEDTK 

108.5 

11 

42.50 

9.9 

COLEDTK 

44.50 

6 

23.50 

7.4 

TOTAL 

153.0 

17 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  0.905 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.3657 

CASES  INCLUDED  17  MISSING  CASES  5 


HSEDTK-TASK  SCORES  FOR  HIGH  SCHOOL  EDUCATION  IN  CBT  GROUP 
COLEDTK-TASK  SCORES  FOR  SOME  COLLEGE  EDUCATION  OR  ABOVE  IN  CBT 
GROUP 
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RANK  SUM  TWO  SAMPLE  TEST  FOR  COMPEX  VS  NCOMEX 


VARIABLE 

RANK  SUM 

SAMPLE 

SIZE 

U  STAT 

AVERAGE 

RANK 

COMPEX 

73.00 

8 

37.00 

9.1 

NCOMEX 

80.00 

9 

35.00 

8.9 

TOTAL 

153.0 

17 

NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  0.048 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.9616 

CASES  INCLUDED  17  MISSING  CASES  5 


COMPEX-TEST  SCORES  FOR  OVER  6  MONTHS  OF  COMPUTER  EXPERIENCE  IN 
CBT  GROUP 

NCOMEX-TEST  SCORES  FOR  NO  OR  LESS  THAN  6  MONTHS  COMPUTER 
EXPERIENCE  IN  CBT  GROUP 


RANK  SUM  TWO  SAMPLE  TEST  FOR  COMPEXK  VS  NCOMEXK 


SAMPLE 

VARIABLE  RANK  SUM  SIZE 


AVERAGE 

U  STAT  RANK 


COMPEXK  80 . 00 
NCOMEXK  73.00 
TOTAL  153.0 


8  44.00  10.0 

9  28.00  8.1 

17 


NORMAL  APPROXIMATION  WITH  CONTINUITY  CORRECTION  0.722 
TWO  TAILED  P  VALUE  FOR  NORMAL  APPROXIMATION  0.4705 


CASES  INCLUDED  17  MISSING  CASES  5 


COMPEXK- TASK  SCORES  FOR  OVER  6  MONTHS  OF  COMPUTER  EXPERIENCE 
IN  CBT  GROUP 

NCOMEXK-TASK  SCORES  FOR  NO  OR  LESS  THAN  6  MONTHS  COMPUTER 
EXPERIENCE  IN  CBT  GROUP 
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